Note: This is test shard 2 of 8. [==========] Running 6 tests from 6 test cases. [----------] Global test environment set-up. [----------] 1 test from RebalanceIgnoredTserversTest [ RUN ] RebalanceIgnoredTserversTest.Basic WARNING: Logging before InitGoogleLogging() is written to STDERR I1121 10:21:10.679478 5400 test_util.cc:225] Using random seed: -1476571236 I1121 10:21:10.685461 5400 ts_itest-base.cc:112] Starting cluster with: I1121 10:21:10.685492 5400 ts_itest-base.cc:113] -------------- I1121 10:21:10.685497 5400 ts_itest-base.cc:114] 5 tablet servers I1121 10:21:10.685505 5400 ts_itest-base.cc:115] 3 replicas per TS I1121 10:21:10.685508 5400 ts_itest-base.cc:116] -------------- 2019-11-21T10:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2019-11-21T10:21:10Z Disabled control of system clock I1121 10:21:10.693748 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:39777 --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true --builtin_ntp_servers=127.5.70.20:45589 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:10.742211 5412 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:10.742386 5412 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:10.743264 5412 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W1121 10:21:10.743280 5412 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:10.743284 5412 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W1121 10:21:10.743288 5412 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I1121 10:21:10.744653 5412 master_main.cc:98] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:45589 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:39777 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.5412 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:10.746714 5412 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:10.747901 5412 hybrid_clock.cc:183] HybridClock initialized: now 1574331670747880 us; error 40 us; skew 500 ppm I1121 10:21:10.748097 5412 fs_manager.cc:261] Metadata directory not provided I1121 10:21:10.748133 5412 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:10.748296 5412 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:10.748318 5412 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:10.749769 5412 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/instance: uuid: "e7a5685f8fa44b3c90e189b566a2649e" format_stamp: "Formatted at 2019-11-21 10:21:10 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:10.750012 5412 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "e7a5685f8fa44b3c90e189b566a2649e" format_stamp: "Formatted at 2019-11-21 10:21:10 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:10.751685 5412 fs_manager.cc:508] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s I1121 10:21:10.751986 5412 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:10.752113 5412 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:10.752934 5412 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:10.752974 5412 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal uuid: "e7a5685f8fa44b3c90e189b566a2649e" format_stamp: "Formatted at 2019-11-21 10:21:10 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:10.753022 5412 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:10.796370 5412 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:10.800607 5412 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.62:39777 I1121 10:21:10.800943 5412 webserver.cc:382] Webserver started at http://127.5.70.62:35627/ using document root and password file I1121 10:21:10.801218 5412 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb I1121 10:21:10.801776 5473 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:10.804621 5473 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e: Bootstrap starting. I1121 10:21:10.805155 5473 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e: No blocks or log segments found. Creating new log. I1121 10:21:10.805454 5473 log.cc:810] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e: Log is configured to *not* fsync() on all Append() calls I1121 10:21:10.806007 5473 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e: No bootstrap required, opened a new log I1121 10:21:10.807775 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master as pid 5412 I1121 10:21:10.807857 5473 raft_consensus.cc:337] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5685f8fa44b3c90e189b566a2649e" member_type: VOTER } I1121 10:21:10.807926 5473 raft_consensus.cc:363] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:10.807946 5473 raft_consensus.cc:702] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a5685f8fa44b3c90e189b566a2649e, State: Initialized, Role: FOLLOWER I1121 10:21:10.808095 5473 consensus_queue.cc:231] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5685f8fa44b3c90e189b566a2649e" member_type: VOTER } I1121 10:21:10.808120 5473 raft_consensus.cc:377] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:21:10.808133 5473 raft_consensus.cc:469] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:21:10.808157 5473 raft_consensus.cc:2905] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:10.808169 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.70.1:0 --local_ip_for_outbound_sockets=127.5.70.1 --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:45589 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1121 10:21:10.808779 5473 raft_consensus.cc:491] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5685f8fa44b3c90e189b566a2649e" member_type: VOTER } I1121 10:21:10.808856 5473 leader_election.cc:310] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a5685f8fa44b3c90e189b566a2649e; no voters: I1121 10:21:10.808974 5473 leader_election.cc:296] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:21:10.809051 5475 raft_consensus.cc:2653] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:10.809196 5475 raft_consensus.cc:664] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [term 1 LEADER]: Becoming Leader. State: Replica: e7a5685f8fa44b3c90e189b566a2649e, State: Running, Role: LEADER I1121 10:21:10.809291 5475 consensus_queue.cc:208] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5685f8fa44b3c90e189b566a2649e" member_type: VOTER } I1121 10:21:10.809674 5475 sys_catalog.cc:340] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a5685f8fa44b3c90e189b566a2649e. Latest consensus state: current_term: 1 leader_uuid: "e7a5685f8fa44b3c90e189b566a2649e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5685f8fa44b3c90e189b566a2649e" member_type: VOTER } } I1121 10:21:10.809706 5473 sys_catalog.cc:340] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "e7a5685f8fa44b3c90e189b566a2649e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5685f8fa44b3c90e189b566a2649e" member_type: VOTER } } I1121 10:21:10.809708 5477 sys_catalog.cc:340] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a5685f8fa44b3c90e189b566a2649e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5685f8fa44b3c90e189b566a2649e" member_type: VOTER } } I1121 10:21:10.810500 5477 sys_catalog.cc:343] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [sys.catalog]: This master's current role is: LEADER I1121 10:21:10.810456 5473 sys_catalog.cc:343] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [sys.catalog]: This master's current role is: LEADER I1121 10:21:10.810652 5473 sys_catalog.cc:428] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [sys.catalog]: configured and running, proceeding with master startup. I1121 10:21:10.810395 5475 sys_catalog.cc:343] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e [sys.catalog]: This master's current role is: LEADER I1121 10:21:10.810741 5481 catalog_manager.cc:1090] Loading table and tablet metadata into memory... I1121 10:21:10.811199 5481 catalog_manager.cc:1101] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:10.866065 5478 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:10.866274 5478 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:10.867413 5478 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:10.867444 5478 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.1 I1121 10:21:10.869153 5478 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:45589 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.5478 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:10.872014 5478 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:10.887476 5478 hybrid_clock.cc:183] HybridClock initialized: now 1574331670887427 us; error 49 us; skew 500 ppm I1121 10:21:10.887720 5478 fs_manager.cc:261] Metadata directory not provided I1121 10:21:10.887768 5478 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:10.887882 5478 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:10.887910 5478 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:10.889569 5478 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "2dd9365c71c54e5d83294b31046c5478" format_stamp: "Formatted at 2019-11-21 10:21:10 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:10.889832 5478 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "2dd9365c71c54e5d83294b31046c5478" format_stamp: "Formatted at 2019-11-21 10:21:10 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:10.894520 5478 fs_manager.cc:508] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s I1121 10:21:10.894951 5478 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:10.895095 5478 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:10.895862 5478 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:10.895978 5478 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal uuid: "2dd9365c71c54e5d83294b31046c5478" format_stamp: "Formatted at 2019-11-21 10:21:10 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:10.896021 5478 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:10.896837 5481 catalog_manager.cc:985] Generated new certificate authority record I1121 10:21:10.897819 5481 catalog_manager.cc:1110] Loading token signing keys... I1121 10:21:10.900738 5481 catalog_manager.cc:4455] T 00000000000000000000000000000000 P e7a5685f8fa44b3c90e189b566a2649e: Generated new TSK 0 I1121 10:21:10.900876 5481 catalog_manager.cc:1120] Initializing in-progress tserver states... I1121 10:21:10.928817 5478 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:10.929224 5478 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:10.929244 5478 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:10.938669 5478 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.1:44363 I1121 10:21:10.939117 5478 webserver.cc:382] Webserver started at http://127.5.70.1:34745/ using document root and password file I1121 10:21:10.939450 5478 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb I1121 10:21:10.943538 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 5478 I1121 10:21:10.943825 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.70.2:0 --local_ip_for_outbound_sockets=127.5.70.2 --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:45589 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1121 10:21:10.949928 5605 heartbeater.cc:322] Connected to a master server at 127.5.70.62:39777 I1121 10:21:10.949987 5605 heartbeater.cc:406] Registering TS with master... I1121 10:21:10.950100 5605 heartbeater.cc:455] Master 127.5.70.62:39777 requested a full tablet report, sending... I1121 10:21:10.950582 5440 ts_manager.cc:192] Registered new tserver with Master: 2dd9365c71c54e5d83294b31046c5478 (127.5.70.1:44363) I1121 10:21:10.951447 5440 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.1:46285 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:11.031577 5609 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:11.031756 5609 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:11.032665 5609 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:11.032696 5609 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.2 I1121 10:21:11.033965 5609 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:45589 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.5609 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.2 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:11.036027 5609 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:11.037168 5609 hybrid_clock.cc:183] HybridClock initialized: now 1574331671037145 us; error 38 us; skew 500 ppm I1121 10:21:11.037330 5609 fs_manager.cc:261] Metadata directory not provided I1121 10:21:11.037362 5609 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:11.037444 5609 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:11.037456 5609 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:11.038493 5609 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "96a9bd7c051344349f90ca7d5b3ff658" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.038669 5609 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "96a9bd7c051344349f90ca7d5b3ff658" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.040084 5609 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:11.040336 5609 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:11.040431 5609 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:11.041157 5609 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:11.041224 5609 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal uuid: "96a9bd7c051344349f90ca7d5b3ff658" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.041271 5609 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:11.084576 5609 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:11.084976 5609 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:11.084997 5609 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:11.093451 5609 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.2:46773 I1121 10:21:11.093827 5609 webserver.cc:382] Webserver started at http://127.5.70.2:40675/ using document root and password file I1121 10:21:11.094087 5609 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb I1121 10:21:11.098541 5730 heartbeater.cc:322] Connected to a master server at 127.5.70.62:39777 I1121 10:21:11.098585 5730 heartbeater.cc:406] Registering TS with master... I1121 10:21:11.098780 5730 heartbeater.cc:455] Master 127.5.70.62:39777 requested a full tablet report, sending... I1121 10:21:11.099159 5440 ts_manager.cc:192] Registered new tserver with Master: 96a9bd7c051344349f90ca7d5b3ff658 (127.5.70.2:46773) I1121 10:21:11.099750 5440 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.2:51781 I1121 10:21:11.100129 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 5609 I1121 10:21:11.100438 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.70.3:0 --local_ip_for_outbound_sockets=127.5.70.3 --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:45589 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:11.143327 5733 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:11.143491 5733 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:11.144311 5733 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:11.144338 5733 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.3 I1121 10:21:11.145568 5733 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:45589 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.5733 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.3 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:11.147655 5733 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:11.148792 5733 hybrid_clock.cc:183] HybridClock initialized: now 1574331671148767 us; error 37 us; skew 500 ppm I1121 10:21:11.148942 5733 fs_manager.cc:261] Metadata directory not provided I1121 10:21:11.148975 5733 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:11.149078 5733 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:11.149101 5733 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:11.150214 5733 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "307e604c914d4271a295bf95f1e37f38" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.150413 5733 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "307e604c914d4271a295bf95f1e37f38" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.151887 5733 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:11.152160 5733 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:11.152313 5733 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:11.153043 5733 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:11.153076 5733 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal uuid: "307e604c914d4271a295bf95f1e37f38" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.153117 5733 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:11.171351 5733 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:11.171751 5733 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:11.171773 5733 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:11.178568 5733 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.3:33605 I1121 10:21:11.178941 5733 webserver.cc:382] Webserver started at http://127.5.70.3:40681/ using document root and password file I1121 10:21:11.179222 5733 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb I1121 10:21:11.183902 5854 heartbeater.cc:322] Connected to a master server at 127.5.70.62:39777 I1121 10:21:11.183949 5854 heartbeater.cc:406] Registering TS with master... I1121 10:21:11.184060 5854 heartbeater.cc:455] Master 127.5.70.62:39777 requested a full tablet report, sending... I1121 10:21:11.184396 5440 ts_manager.cc:192] Registered new tserver with Master: 307e604c914d4271a295bf95f1e37f38 (127.5.70.3:33605) I1121 10:21:11.184604 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 5733 I1121 10:21:11.185040 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.70.4:0 --local_ip_for_outbound_sockets=127.5.70.4 --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:45589 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1121 10:21:11.185194 5440 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.3:38643 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:11.227121 5857 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:11.227277 5857 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:11.228057 5857 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:11.228082 5857 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.4 I1121 10:21:11.229286 5857 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:45589 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.5857 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.4 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:11.231225 5857 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:11.232458 5857 hybrid_clock.cc:183] HybridClock initialized: now 1574331671232437 us; error 32 us; skew 500 ppm I1121 10:21:11.232633 5857 fs_manager.cc:261] Metadata directory not provided I1121 10:21:11.232666 5857 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:11.232772 5857 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:11.232795 5857 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:11.233844 5857 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.234045 5857 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.235527 5857 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s I1121 10:21:11.235812 5857 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:11.235931 5857 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:11.236706 5857 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:11.236744 5857 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.236841 5857 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:11.319499 5857 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:11.319875 5857 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:11.319895 5857 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:11.326453 5857 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.4:38301 I1121 10:21:11.326762 5857 webserver.cc:382] Webserver started at http://127.5.70.4:43339/ using document root and password file I1121 10:21:11.327019 5857 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb I1121 10:21:11.329928 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 5857 I1121 10:21:11.330195 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.70.5:0 --local_ip_for_outbound_sockets=127.5.70.5 --webserver_interface=127.5.70.5 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:45589 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I1121 10:21:11.331651 5978 heartbeater.cc:322] Connected to a master server at 127.5.70.62:39777 I1121 10:21:11.331708 5978 heartbeater.cc:406] Registering TS with master... I1121 10:21:11.331892 5978 heartbeater.cc:455] Master 127.5.70.62:39777 requested a full tablet report, sending... I1121 10:21:11.332242 5440 ts_manager.cc:192] Registered new tserver with Master: ad4149bfd9f141b1aebb5ce1c816ecbd (127.5.70.4:38301) I1121 10:21:11.332862 5440 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.4:37449 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:11.372395 5981 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:11.372582 5981 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:11.373389 5981 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:11.373420 5981 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.5 I1121 10:21:11.374749 5981 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:45589 --ntp_initial_sync_wait_secs=10 --time_source=builtin --log_cache_size_limit_mb=10 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.70.5 --webserver_port=0 --tserver_master_addrs=127.5.70.62:39777 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.5981 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.5 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:11.376683 5981 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:11.377827 5981 hybrid_clock.cc:183] HybridClock initialized: now 1574331671377767 us; error 69 us; skew 500 ppm I1121 10:21:11.377969 5981 fs_manager.cc:261] Metadata directory not provided I1121 10:21:11.378006 5981 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:11.378104 5981 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:11.378125 5981 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:11.379228 5981 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "092c2d28165a41f4addecdd85574691e" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.379451 5981 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "092c2d28165a41f4addecdd85574691e" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.380995 5981 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:11.381338 5981 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:11.381513 5981 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:11.382213 5981 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:11.382251 5981 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal uuid: "092c2d28165a41f4addecdd85574691e" format_stamp: "Formatted at 2019-11-21 10:21:11 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:11.382311 5981 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:11.425110 5981 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:11.425500 5981 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:11.425519 5981 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:11.432962 5981 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.5:34397 I1121 10:21:11.433317 5981 webserver.cc:382] Webserver started at http://127.5.70.5:35897/ using document root and password file I1121 10:21:11.433583 5981 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb I1121 10:21:11.434793 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 5981 I1121 10:21:11.438081 6102 heartbeater.cc:322] Connected to a master server at 127.5.70.62:39777 I1121 10:21:11.438141 6102 heartbeater.cc:406] Registering TS with master... I1121 10:21:11.438273 6102 heartbeater.cc:455] Master 127.5.70.62:39777 requested a full tablet report, sending... I1121 10:21:11.438657 5440 ts_manager.cc:192] Registered new tserver with Master: 092c2d28165a41f4addecdd85574691e (127.5.70.5:34397) I1121 10:21:11.439147 5440 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.5:52871 I1121 10:21:11.440615 5400 external_mini_cluster.cc:717] 5 TS(s) registered with all masters I1121 10:21:11.447556 5440 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34062: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I1121 10:21:11.453626 5915 tablet_service.cc:1023] Processing CreateTablet for tablet 43430bbf87f647d98bdd8449ab706c8a (table=TestTable [id=f71508525d1f4e8b81e6f570c8b3ddbb]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:11.453853 5527 tablet_service.cc:1023] Processing CreateTablet for tablet 43430bbf87f647d98bdd8449ab706c8a (table=TestTable [id=f71508525d1f4e8b81e6f570c8b3ddbb]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:11.453864 5915 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 43430bbf87f647d98bdd8449ab706c8a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:11.453977 5527 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 43430bbf87f647d98bdd8449ab706c8a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:11.453976 6039 tablet_service.cc:1023] Processing CreateTablet for tablet 43430bbf87f647d98bdd8449ab706c8a (table=TestTable [id=f71508525d1f4e8b81e6f570c8b3ddbb]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:11.454139 6039 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 43430bbf87f647d98bdd8449ab706c8a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:11.457258 6121 tablet_bootstrap.cc:439] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478: Bootstrap starting. I1121 10:21:11.457253 6123 tablet_bootstrap.cc:439] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e: Bootstrap starting. I1121 10:21:11.457760 6122 tablet_bootstrap.cc:439] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd: Bootstrap starting. I1121 10:21:11.457979 6121 tablet_bootstrap.cc:590] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478: No blocks or log segments found. Creating new log. I1121 10:21:11.458369 6121 log.cc:810] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478: Log is configured to *not* fsync() on all Append() calls I1121 10:21:11.458384 6122 tablet_bootstrap.cc:590] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd: No blocks or log segments found. Creating new log. I1121 10:21:11.458729 6122 log.cc:810] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd: Log is configured to *not* fsync() on all Append() calls I1121 10:21:11.459089 6123 tablet_bootstrap.cc:590] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e: No blocks or log segments found. Creating new log. I1121 10:21:11.459337 6123 log.cc:810] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e: Log is configured to *not* fsync() on all Append() calls I1121 10:21:11.459930 6121 tablet_bootstrap.cc:439] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478: No bootstrap required, opened a new log I1121 10:21:11.459990 6121 ts_tablet_manager.cc:1104] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:11.460074 6123 tablet_bootstrap.cc:439] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e: No bootstrap required, opened a new log I1121 10:21:11.460134 6123 ts_tablet_manager.cc:1104] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:11.460245 6122 tablet_bootstrap.cc:439] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd: No bootstrap required, opened a new log I1121 10:21:11.460328 6122 ts_tablet_manager.cc:1104] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:11.461992 6123 raft_consensus.cc:337] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } } I1121 10:21:11.462043 6123 raft_consensus.cc:363] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:11.462060 6123 raft_consensus.cc:702] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092c2d28165a41f4addecdd85574691e, State: Initialized, Role: FOLLOWER I1121 10:21:11.462121 6121 raft_consensus.cc:337] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } } I1121 10:21:11.462209 6123 consensus_queue.cc:231] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } } I1121 10:21:11.462227 6121 raft_consensus.cc:363] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:11.462250 6121 raft_consensus.cc:702] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd9365c71c54e5d83294b31046c5478, State: Initialized, Role: FOLLOWER I1121 10:21:11.462384 6122 raft_consensus.cc:337] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } } I1121 10:21:11.462419 6102 heartbeater.cc:447] Master 127.5.70.62:39777 was elected leader, sending a full tablet report... I1121 10:21:11.462425 6121 consensus_queue.cc:231] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } } I1121 10:21:11.462443 6123 ts_tablet_manager.cc:1132] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I1121 10:21:11.462599 5605 heartbeater.cc:447] Master 127.5.70.62:39777 was elected leader, sending a full tablet report... I1121 10:21:11.462605 6121 ts_tablet_manager.cc:1132] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s I1121 10:21:11.462657 6122 raft_consensus.cc:363] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:11.462683 6122 raft_consensus.cc:702] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4149bfd9f141b1aebb5ce1c816ecbd, State: Initialized, Role: FOLLOWER I1121 10:21:11.462834 6122 consensus_queue.cc:231] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } } I1121 10:21:11.463034 5978 heartbeater.cc:447] Master 127.5.70.62:39777 was elected leader, sending a full tablet report... I1121 10:21:11.463049 6122 ts_tablet_manager.cc:1132] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s W1121 10:21:11.577651 5979 tablet.cc:1767] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd: Can't schedule compaction. Clean time has not been advanced past its initial value. W1121 10:21:11.684187 6103 tablet.cc:1767] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e: Can't schedule compaction. Clean time has not been advanced past its initial value. W1121 10:21:11.691493 5606 tablet.cc:1767] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:11.823639 6127 raft_consensus.cc:469] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:11.823812 6127 raft_consensus.cc:491] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } } I1121 10:21:11.824033 6127 leader_election.cc:296] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2dd9365c71c54e5d83294b31046c5478 (127.5.70.1:44363), ad4149bfd9f141b1aebb5ce1c816ecbd (127.5.70.4:38301) I1121 10:21:11.827190 5935 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "43430bbf87f647d98bdd8449ab706c8a" candidate_uuid: "092c2d28165a41f4addecdd85574691e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" is_pre_election: true I1121 10:21:11.827327 5935 raft_consensus.cc:2359] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 092c2d28165a41f4addecdd85574691e in term 0. I1121 10:21:11.827440 5562 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "43430bbf87f647d98bdd8449ab706c8a" candidate_uuid: "092c2d28165a41f4addecdd85574691e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd9365c71c54e5d83294b31046c5478" is_pre_election: true I1121 10:21:11.827519 5562 raft_consensus.cc:2359] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 092c2d28165a41f4addecdd85574691e in term 0. I1121 10:21:11.827558 5990 leader_election.cc:310] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 092c2d28165a41f4addecdd85574691e, ad4149bfd9f141b1aebb5ce1c816ecbd; no voters: I1121 10:21:11.827634 6127 raft_consensus.cc:2653] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:11.827672 6127 raft_consensus.cc:469] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:11.827682 6127 raft_consensus.cc:2905] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:11.828424 6127 raft_consensus.cc:491] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } } I1121 10:21:11.828503 6127 leader_election.cc:296] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [CANDIDATE]: Term 1 election: Requested vote from peers 2dd9365c71c54e5d83294b31046c5478 (127.5.70.1:44363), ad4149bfd9f141b1aebb5ce1c816ecbd (127.5.70.4:38301) I1121 10:21:11.828693 5562 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "43430bbf87f647d98bdd8449ab706c8a" candidate_uuid: "092c2d28165a41f4addecdd85574691e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dd9365c71c54e5d83294b31046c5478" I1121 10:21:11.828730 5935 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "43430bbf87f647d98bdd8449ab706c8a" candidate_uuid: "092c2d28165a41f4addecdd85574691e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" I1121 10:21:11.828728 5562 raft_consensus.cc:2905] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:11.828749 5935 raft_consensus.cc:2905] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:11.829382 5935 raft_consensus.cc:2359] T 43430bbf87f647d98bdd8449ab706c8a P ad4149bfd9f141b1aebb5ce1c816ecbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 092c2d28165a41f4addecdd85574691e in term 1. I1121 10:21:11.829489 5562 raft_consensus.cc:2359] T 43430bbf87f647d98bdd8449ab706c8a P 2dd9365c71c54e5d83294b31046c5478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 092c2d28165a41f4addecdd85574691e in term 1. I1121 10:21:11.829526 5990 leader_election.cc:310] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 092c2d28165a41f4addecdd85574691e, ad4149bfd9f141b1aebb5ce1c816ecbd; no voters: I1121 10:21:11.829599 6127 raft_consensus.cc:2653] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:11.829728 6127 raft_consensus.cc:664] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [term 1 LEADER]: Becoming Leader. State: Replica: 092c2d28165a41f4addecdd85574691e, State: Running, Role: LEADER I1121 10:21:11.829810 6127 consensus_queue.cc:208] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } } I1121 10:21:11.830701 5440 catalog_manager.cc:4143] T 43430bbf87f647d98bdd8449ab706c8a P 092c2d28165a41f4addecdd85574691e reported cstate change: term changed from 0 to 1, leader changed from to 092c2d28165a41f4addecdd85574691e (127.5.70.5). New cstate: current_term: 1 leader_uuid: "092c2d28165a41f4addecdd85574691e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dd9365c71c54e5d83294b31046c5478" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad4149bfd9f141b1aebb5ce1c816ecbd" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 38301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "092c2d28165a41f4addecdd85574691e" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 34397 } health_report { overall_health: HEALTHY } } } I1121 10:21:11.891585 5400 external_mini_cluster.cc:717] 5 TS(s) registered with all masters I1121 10:21:11.892356 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2dd9365c71c54e5d83294b31046c5478 to finish bootstrapping I1121 10:21:11.895556 5400 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 96a9bd7c051344349f90ca7d5b3ff658 to finish bootstrapping I1121 10:21:11.898411 5400 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 307e604c914d4271a295bf95f1e37f38 to finish bootstrapping I1121 10:21:11.901356 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ad4149bfd9f141b1aebb5ce1c816ecbd to finish bootstrapping I1121 10:21:11.903820 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 092c2d28165a41f4addecdd85574691e to finish bootstrapping I1121 10:21:12.100630 5730 heartbeater.cc:447] Master 127.5.70.62:39777 was elected leader, sending a full tablet report... I1121 10:21:12.142050 5440 ts_manager.cc:278] Set tserver state for 2dd9365c71c54e5d83294b31046c5478 to MAINTENANCE_MODE I1121 10:21:12.186054 5854 heartbeater.cc:447] Master 127.5.70.62:39777 was elected leader, sending a full tablet report... /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/rebalancer_tool-test.cc:350: Failure Value of: out Expected: has substring "2dd9365c71c54e5d83294b31046c5478 | 0" Actual: "Per-server replica distribution summary for tservers_to_empty:\n Server UUID | Replica Count\n----------------------------------+---------------\n 2dd9365c71c54e5d83294b31046c5478 | 1\n\nPer-server replica distribution summary:\n Statistic | Value\n-----------------------+----------\n Minimum Replica Count | 0\n Maximum Replica Count | 1\n Average Replica Count | 0.500000\n\nPer-table replica distribution summary:\n Replica Skew | Value\n--------------+----------\n Minimum | 1\n Maximum | 1\n Average | 1.000000\n\n\nrebalancing is complete: cluster is balanced (moved 0 replicas)\n" (of type std::string) stderr: W1121 10:21:12.202622 6239 flags.cc:404] Enabled unsafe flag: --never_fsync=true I1121 10:21:12.270890 6239 rebalancer_tool.cc:190] replacing replicas on healthy ignored tservers I1121 10:21:12.299615 6239 rebalancer_tool.cc:199] running whole-cluster rebalancing I1121 10:21:12.331624 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 5478 I1121 10:21:12.334978 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 5609 I1121 10:21:12.338331 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 5733 I1121 10:21:12.341122 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 5857 I1121 10:21:12.344296 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 5981 I1121 10:21:12.347128 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master with pid 5412 2019-11-21T10:21:12Z chronyd exiting I1121 10:21:12.361392 5400 test_util.cc:136] ----------------------------------------------- I1121 10:21:12.361430 5400 test_util.cc:137] Had fatal failures, leaving test files at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1574331670674946-5400 [ FAILED ] RebalanceIgnoredTserversTest.Basic (1683 ms) [----------] 1 test from RebalanceIgnoredTserversTest (1683 ms total) [----------] 1 test from RebalanceStartSafetyTest [ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 I1121 10:21:12.361819 5400 test_util.cc:225] Using random seed: -1474888871 I1121 10:21:12.362223 5400 ts_itest-base.cc:112] Starting cluster with: I1121 10:21:12.362246 5400 ts_itest-base.cc:113] -------------- I1121 10:21:12.362251 5400 ts_itest-base.cc:114] 5 tablet servers I1121 10:21:12.362255 5400 ts_itest-base.cc:115] 3 replicas per TS I1121 10:21:12.362259 5400 ts_itest-base.cc:116] -------------- 2019-11-21T10:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2019-11-21T10:21:12Z Disabled control of system clock I1121 10:21:12.369056 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:34769 --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true --builtin_ntp_servers=127.5.70.20:46351 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:12.413535 6368 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:12.413707 6368 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:12.414614 6368 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:12.414636 6368 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W1121 10:21:12.414644 6368 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:12.414649 6368 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W1121 10:21:12.414661 6368 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I1121 10:21:12.416002 6368 master_main.cc:98] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:46351 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:34769 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.6368 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:12.417778 6368 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:12.418967 6368 hybrid_clock.cc:183] HybridClock initialized: now 1574331672418918 us; error 63 us; skew 500 ppm I1121 10:21:12.419124 6368 fs_manager.cc:261] Metadata directory not provided I1121 10:21:12.419155 6368 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:12.419253 6368 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:12.419273 6368 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:12.420346 6368 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/instance: uuid: "c8ca76c9b6904748aab67c5cd63dd418" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.420568 6368 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "c8ca76c9b6904748aab67c5cd63dd418" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.422041 6368 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s I1121 10:21:12.422291 6368 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:12.422387 6368 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:12.423022 6368 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:12.423055 6368 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal uuid: "c8ca76c9b6904748aab67c5cd63dd418" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.423092 6368 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:12.449306 6368 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:12.452064 6368 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.62:34769 I1121 10:21:12.452340 6368 webserver.cc:382] Webserver started at http://127.5.70.62:34527/ using document root and password file I1121 10:21:12.452837 6368 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb I1121 10:21:12.453339 6429 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:12.455876 6429 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418: Bootstrap starting. I1121 10:21:12.456343 6429 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418: No blocks or log segments found. Creating new log. I1121 10:21:12.456607 6429 log.cc:810] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418: Log is configured to *not* fsync() on all Append() calls I1121 10:21:12.457101 6429 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418: No bootstrap required, opened a new log I1121 10:21:12.458366 6429 raft_consensus.cc:337] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ca76c9b6904748aab67c5cd63dd418" member_type: VOTER } I1121 10:21:12.458423 6429 raft_consensus.cc:363] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:12.458452 6429 raft_consensus.cc:702] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ca76c9b6904748aab67c5cd63dd418, State: Initialized, Role: FOLLOWER I1121 10:21:12.458591 6429 consensus_queue.cc:231] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ca76c9b6904748aab67c5cd63dd418" member_type: VOTER } I1121 10:21:12.458621 6429 raft_consensus.cc:377] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:21:12.458638 6429 raft_consensus.cc:469] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:21:12.458670 6429 raft_consensus.cc:2905] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:12.459132 6429 raft_consensus.cc:491] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ca76c9b6904748aab67c5cd63dd418" member_type: VOTER } I1121 10:21:12.459226 6429 leader_election.cc:310] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8ca76c9b6904748aab67c5cd63dd418; no voters: I1121 10:21:12.459336 6429 leader_election.cc:296] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:21:12.459484 6431 raft_consensus.cc:2653] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:12.459491 6429 sys_catalog.cc:340] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ca76c9b6904748aab67c5cd63dd418" member_type: VOTER } } I1121 10:21:12.459534 6429 sys_catalog.cc:343] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [sys.catalog]: This master's current role is: FOLLOWER I1121 10:21:12.459606 6431 raft_consensus.cc:664] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [term 1 LEADER]: Becoming Leader. State: Replica: c8ca76c9b6904748aab67c5cd63dd418, State: Running, Role: LEADER I1121 10:21:12.459678 6431 consensus_queue.cc:208] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ca76c9b6904748aab67c5cd63dd418" member_type: VOTER } I1121 10:21:12.459882 6429 sys_catalog.cc:428] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [sys.catalog]: configured and running, proceeding with master startup. I1121 10:21:12.459906 6431 sys_catalog.cc:340] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8ca76c9b6904748aab67c5cd63dd418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ca76c9b6904748aab67c5cd63dd418" member_type: VOTER } } I1121 10:21:12.459973 6431 sys_catalog.cc:343] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [sys.catalog]: This master's current role is: LEADER I1121 10:21:12.460057 6432 sys_catalog.cc:340] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ca76c9b6904748aab67c5cd63dd418. Latest consensus state: current_term: 1 leader_uuid: "c8ca76c9b6904748aab67c5cd63dd418" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ca76c9b6904748aab67c5cd63dd418" member_type: VOTER } } I1121 10:21:12.460383 6432 sys_catalog.cc:343] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418 [sys.catalog]: This master's current role is: LEADER I1121 10:21:12.460198 6436 catalog_manager.cc:1090] Loading table and tablet metadata into memory... I1121 10:21:12.460816 6436 catalog_manager.cc:1101] Initializing Kudu internal certificate authority... I1121 10:21:12.462447 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master as pid 6368 I1121 10:21:12.462769 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.70.1:0 --local_ip_for_outbound_sockets=127.5.70.1 --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:46351 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:12.505614 6439 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:12.505790 6439 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:12.506593 6439 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:12.506614 6439 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:12.506633 6439 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.1 I1121 10:21:12.507760 6439 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:46351 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.6439 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:12.509676 6439 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:12.515987 6436 catalog_manager.cc:985] Generated new certificate authority record I1121 10:21:12.516665 6436 catalog_manager.cc:1110] Loading token signing keys... I1121 10:21:12.517082 6439 hybrid_clock.cc:183] HybridClock initialized: now 1574331672517056 us; error 33 us; skew 500 ppm I1121 10:21:12.517269 6439 fs_manager.cc:261] Metadata directory not provided I1121 10:21:12.517310 6439 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:12.517408 6439 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:12.517453 6439 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:12.518771 6439 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "6f5b7acb7f9d4a6389c4ce907c928f7e" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.518980 6439 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "6f5b7acb7f9d4a6389c4ce907c928f7e" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.521394 6436 catalog_manager.cc:4455] T 00000000000000000000000000000000 P c8ca76c9b6904748aab67c5cd63dd418: Generated new TSK 0 I1121 10:21:12.521502 6436 catalog_manager.cc:1120] Initializing in-progress tserver states... I1121 10:21:12.524916 6439 fs_manager.cc:508] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s I1121 10:21:12.525243 6439 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:12.525393 6439 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:12.526131 6439 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:12.526171 6439 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal uuid: "6f5b7acb7f9d4a6389c4ce907c928f7e" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.526258 6439 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:12.561843 6439 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:12.562187 6439 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:12.562206 6439 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:12.569438 6439 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.1:43875 I1121 10:21:12.569756 6439 webserver.cc:382] Webserver started at http://127.5.70.1:40079/ using document root and password file I1121 10:21:12.570016 6439 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb I1121 10:21:12.577610 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 6439 I1121 10:21:12.577908 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.70.2:0 --local_ip_for_outbound_sockets=127.5.70.2 --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:46351 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:21:12.582579 6561 heartbeater.cc:322] Connected to a master server at 127.5.70.62:34769 I1121 10:21:12.582643 6561 heartbeater.cc:406] Registering TS with master... I1121 10:21:12.582819 6561 heartbeater.cc:455] Master 127.5.70.62:34769 requested a full tablet report, sending... I1121 10:21:12.583387 6396 ts_manager.cc:192] Registered new tserver with Master: 6f5b7acb7f9d4a6389c4ce907c928f7e (127.5.70.1:43875) I1121 10:21:12.584331 6396 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.1:35011 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:12.642006 6565 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:12.642159 6565 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:12.643035 6565 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:12.643056 6565 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:12.643069 6565 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.2 I1121 10:21:12.644265 6565 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:46351 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.6565 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.2 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:12.646082 6565 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:12.647220 6565 hybrid_clock.cc:183] HybridClock initialized: now 1574331672647191 us; error 46 us; skew 500 ppm I1121 10:21:12.647358 6565 fs_manager.cc:261] Metadata directory not provided I1121 10:21:12.647394 6565 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:12.647506 6565 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:12.647531 6565 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:12.648613 6565 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "05aeabf2d12f489da2a93e862bb57760" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.648810 6565 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "05aeabf2d12f489da2a93e862bb57760" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.650228 6565 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:21:12.650508 6565 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:12.650629 6565 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:12.651329 6565 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:12.651366 6565 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal uuid: "05aeabf2d12f489da2a93e862bb57760" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.651415 6565 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:12.679047 6565 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:12.679409 6565 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:12.679427 6565 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:12.684991 6565 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.2:34929 I1121 10:21:12.685328 6565 webserver.cc:382] Webserver started at http://127.5.70.2:43127/ using document root and password file I1121 10:21:12.685611 6565 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb I1121 10:21:12.690055 6686 heartbeater.cc:322] Connected to a master server at 127.5.70.62:34769 I1121 10:21:12.690102 6686 heartbeater.cc:406] Registering TS with master... I1121 10:21:12.690201 6686 heartbeater.cc:455] Master 127.5.70.62:34769 requested a full tablet report, sending... I1121 10:21:12.690526 6396 ts_manager.cc:192] Registered new tserver with Master: 05aeabf2d12f489da2a93e862bb57760 (127.5.70.2:34929) I1121 10:21:12.691051 6396 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.2:45537 I1121 10:21:12.694205 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 6565 I1121 10:21:12.694510 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.70.3:0 --local_ip_for_outbound_sockets=127.5.70.3 --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:46351 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:12.732008 6689 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:12.732173 6689 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:12.733003 6689 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:12.733026 6689 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:12.733040 6689 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.3 I1121 10:21:12.734158 6689 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:46351 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.6689 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.3 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:12.735868 6689 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:12.736991 6689 hybrid_clock.cc:183] HybridClock initialized: now 1574331672736968 us; error 33 us; skew 500 ppm I1121 10:21:12.737134 6689 fs_manager.cc:261] Metadata directory not provided I1121 10:21:12.737164 6689 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:12.737268 6689 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:12.737288 6689 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:12.738287 6689 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "c86302eb47f945cf8b90bc1c62a703c5" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.738477 6689 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "c86302eb47f945cf8b90bc1c62a703c5" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.739868 6689 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I1121 10:21:12.740183 6689 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:12.740314 6689 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:12.741008 6689 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:12.741044 6689 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal uuid: "c86302eb47f945cf8b90bc1c62a703c5" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.741097 6689 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:12.774556 6689 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:12.775027 6689 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:12.775045 6689 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:12.780143 6689 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.3:44189 I1121 10:21:12.780464 6689 webserver.cc:382] Webserver started at http://127.5.70.3:42997/ using document root and password file I1121 10:21:12.780781 6689 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb I1121 10:21:12.784910 6810 heartbeater.cc:322] Connected to a master server at 127.5.70.62:34769 I1121 10:21:12.784955 6810 heartbeater.cc:406] Registering TS with master... I1121 10:21:12.785025 6810 heartbeater.cc:455] Master 127.5.70.62:34769 requested a full tablet report, sending... I1121 10:21:12.785332 6396 ts_manager.cc:192] Registered new tserver with Master: c86302eb47f945cf8b90bc1c62a703c5 (127.5.70.3:44189) I1121 10:21:12.785800 6396 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.3:36497 I1121 10:21:12.788638 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 6689 I1121 10:21:12.788868 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.70.4:0 --local_ip_for_outbound_sockets=127.5.70.4 --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:46351 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:12.825482 6813 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:12.825628 6813 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:12.826377 6813 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:12.826395 6813 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:12.826407 6813 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.4 I1121 10:21:12.827576 6813 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:46351 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.6813 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.4 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:12.829341 6813 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:12.830479 6813 hybrid_clock.cc:183] HybridClock initialized: now 1574331672830458 us; error 36 us; skew 500 ppm I1121 10:21:12.830636 6813 fs_manager.cc:261] Metadata directory not provided I1121 10:21:12.830669 6813 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:12.830827 6813 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:12.830850 6813 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:12.831982 6813 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "17a86b7e343646309c0acbd83b846e4d" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.832181 6813 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "17a86b7e343646309c0acbd83b846e4d" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.833634 6813 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:21:12.833947 6813 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:12.834062 6813 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:12.834709 6813 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:12.834772 6813 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal uuid: "17a86b7e343646309c0acbd83b846e4d" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.834805 6813 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:12.852326 6813 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:12.852747 6813 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:12.852767 6813 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:12.858739 6813 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.4:35017 I1121 10:21:12.859067 6813 webserver.cc:382] Webserver started at http://127.5.70.4:45825/ using document root and password file I1121 10:21:12.859333 6813 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb I1121 10:21:12.862565 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 6813 I1121 10:21:12.862958 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.70.5:0 --local_ip_for_outbound_sockets=127.5.70.5 --webserver_interface=127.5.70.5 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:46351 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:21:12.863601 6934 heartbeater.cc:322] Connected to a master server at 127.5.70.62:34769 I1121 10:21:12.863639 6934 heartbeater.cc:406] Registering TS with master... I1121 10:21:12.863718 6934 heartbeater.cc:455] Master 127.5.70.62:34769 requested a full tablet report, sending... I1121 10:21:12.864068 6396 ts_manager.cc:192] Registered new tserver with Master: 17a86b7e343646309c0acbd83b846e4d (127.5.70.4:35017) I1121 10:21:12.864670 6396 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.4:44247 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:12.899871 6937 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:12.900039 6937 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:12.900924 6937 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:12.900943 6937 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:12.900956 6937 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.5 I1121 10:21:12.902155 6937 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:46351 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.70.5 --webserver_port=0 --tserver_master_addrs=127.5.70.62:34769 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.6937 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.5 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:12.903954 6937 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:12.905088 6937 hybrid_clock.cc:183] HybridClock initialized: now 1574331672905071 us; error 40 us; skew 500 ppm I1121 10:21:12.905234 6937 fs_manager.cc:261] Metadata directory not provided I1121 10:21:12.905272 6937 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:12.905378 6937 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:12.905398 6937 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:12.906442 6937 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "fd71d3e0796243de9ac2a8ca6d433918" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.906642 6937 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "fd71d3e0796243de9ac2a8ca6d433918" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.908030 6937 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s I1121 10:21:12.908324 6937 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:12.908444 6937 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:12.909127 6937 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:12.909162 6937 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal uuid: "fd71d3e0796243de9ac2a8ca6d433918" format_stamp: "Formatted at 2019-11-21 10:21:12 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:12.909229 6937 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:12.927700 6937 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:12.928215 6937 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:12.928234 6937 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:12.933573 6937 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.5:40777 I1121 10:21:12.933871 6937 webserver.cc:382] Webserver started at http://127.5.70.5:38723/ using document root and password file I1121 10:21:12.934119 6937 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb I1121 10:21:12.936921 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 6937 I1121 10:21:12.938393 7058 heartbeater.cc:322] Connected to a master server at 127.5.70.62:34769 I1121 10:21:12.938448 7058 heartbeater.cc:406] Registering TS with master... I1121 10:21:12.938603 7058 heartbeater.cc:455] Master 127.5.70.62:34769 requested a full tablet report, sending... I1121 10:21:12.939034 6396 ts_manager.cc:192] Registered new tserver with Master: fd71d3e0796243de9ac2a8ca6d433918 (127.5.70.5:40777) I1121 10:21:12.939545 6396 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.5:58497 I1121 10:21:12.940137 5400 external_mini_cluster.cc:717] 5 TS(s) registered with all masters I1121 10:21:12.945154 6396 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58928: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I1121 10:21:12.950755 6747 tablet_service.cc:1023] Processing CreateTablet for tablet 028c77fc16a1479d859f65a3b4a3fb98 (table=TestTable [id=8efb8b7ec6c445998cd135a0133d43ee]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:12.950947 6747 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 028c77fc16a1479d859f65a3b4a3fb98. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:12.951653 6618 tablet_service.cc:1023] Processing CreateTablet for tablet 028c77fc16a1479d859f65a3b4a3fb98 (table=TestTable [id=8efb8b7ec6c445998cd135a0133d43ee]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:12.951856 6618 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 028c77fc16a1479d859f65a3b4a3fb98. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:12.953194 7074 tablet_bootstrap.cc:439] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5: Bootstrap starting. I1121 10:21:12.953704 7074 tablet_bootstrap.cc:590] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5: No blocks or log segments found. Creating new log. I1121 10:21:12.953966 7074 log.cc:810] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5: Log is configured to *not* fsync() on all Append() calls I1121 10:21:12.954097 6995 tablet_service.cc:1023] Processing CreateTablet for tablet 028c77fc16a1479d859f65a3b4a3fb98 (table=TestTable [id=8efb8b7ec6c445998cd135a0133d43ee]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:12.954255 7075 tablet_bootstrap.cc:439] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760: Bootstrap starting. I1121 10:21:12.954277 6995 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 028c77fc16a1479d859f65a3b4a3fb98. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:12.954701 7075 tablet_bootstrap.cc:590] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760: No blocks or log segments found. Creating new log. I1121 10:21:12.954948 7075 log.cc:810] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760: Log is configured to *not* fsync() on all Append() calls I1121 10:21:12.955593 7074 tablet_bootstrap.cc:439] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5: No bootstrap required, opened a new log I1121 10:21:12.955652 7074 ts_tablet_manager.cc:1104] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:12.955684 7075 tablet_bootstrap.cc:439] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760: No bootstrap required, opened a new log I1121 10:21:12.955739 7075 ts_tablet_manager.cc:1104] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:12.957048 7074 raft_consensus.cc:337] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } } I1121 10:21:12.957108 7074 raft_consensus.cc:363] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:12.957130 7074 raft_consensus.cc:702] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86302eb47f945cf8b90bc1c62a703c5, State: Initialized, Role: FOLLOWER I1121 10:21:12.957274 7074 consensus_queue.cc:231] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } } I1121 10:21:12.957417 7075 raft_consensus.cc:337] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } } I1121 10:21:12.957449 7074 ts_tablet_manager.cc:1132] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I1121 10:21:12.957453 6810 heartbeater.cc:447] Master 127.5.70.62:34769 was elected leader, sending a full tablet report... I1121 10:21:12.957474 7075 raft_consensus.cc:363] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:12.957494 7075 raft_consensus.cc:702] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05aeabf2d12f489da2a93e862bb57760, State: Initialized, Role: FOLLOWER I1121 10:21:12.957612 7075 consensus_queue.cc:231] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } } I1121 10:21:12.957764 6686 heartbeater.cc:447] Master 127.5.70.62:34769 was elected leader, sending a full tablet report... I1121 10:21:12.957782 7075 ts_tablet_manager.cc:1132] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I1121 10:21:12.957911 7078 tablet_bootstrap.cc:439] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918: Bootstrap starting. I1121 10:21:12.958395 7078 tablet_bootstrap.cc:590] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918: No blocks or log segments found. Creating new log. I1121 10:21:12.958642 7078 log.cc:810] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918: Log is configured to *not* fsync() on all Append() calls I1121 10:21:12.959410 7078 tablet_bootstrap.cc:439] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918: No bootstrap required, opened a new log I1121 10:21:12.959447 7078 ts_tablet_manager.cc:1104] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:12.960702 7078 raft_consensus.cc:337] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } } I1121 10:21:12.960747 7078 raft_consensus.cc:363] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:12.960757 7078 raft_consensus.cc:702] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd71d3e0796243de9ac2a8ca6d433918, State: Initialized, Role: FOLLOWER I1121 10:21:12.960857 7078 consensus_queue.cc:231] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } } I1121 10:21:12.960999 7058 heartbeater.cc:447] Master 127.5.70.62:34769 was elected leader, sending a full tablet report... I1121 10:21:12.961000 7078 ts_tablet_manager.cc:1132] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s W1121 10:21:13.031301 6811 tablet.cc:1767] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:13.066846 7079 raft_consensus.cc:469] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:13.066974 7079 raft_consensus.cc:491] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } } I1121 10:21:13.067147 7079 leader_election.cc:296] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd71d3e0796243de9ac2a8ca6d433918 (127.5.70.5:40777), 05aeabf2d12f489da2a93e862bb57760 (127.5.70.2:34929) I1121 10:21:13.070067 7015 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "028c77fc16a1479d859f65a3b4a3fb98" candidate_uuid: "c86302eb47f945cf8b90bc1c62a703c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd71d3e0796243de9ac2a8ca6d433918" is_pre_election: true I1121 10:21:13.070104 6643 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "028c77fc16a1479d859f65a3b4a3fb98" candidate_uuid: "c86302eb47f945cf8b90bc1c62a703c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aeabf2d12f489da2a93e862bb57760" is_pre_election: true I1121 10:21:13.070156 7015 raft_consensus.cc:2359] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86302eb47f945cf8b90bc1c62a703c5 in term 0. I1121 10:21:13.070156 6643 raft_consensus.cc:2359] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86302eb47f945cf8b90bc1c62a703c5 in term 0. I1121 10:21:13.070336 6700 leader_election.cc:310] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05aeabf2d12f489da2a93e862bb57760, c86302eb47f945cf8b90bc1c62a703c5; no voters: I1121 10:21:13.070432 7079 raft_consensus.cc:2653] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:13.070470 7079 raft_consensus.cc:469] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:13.070480 7079 raft_consensus.cc:2905] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:13.071076 7079 raft_consensus.cc:491] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } } I1121 10:21:13.071137 7079 leader_election.cc:296] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [CANDIDATE]: Term 1 election: Requested vote from peers fd71d3e0796243de9ac2a8ca6d433918 (127.5.70.5:40777), 05aeabf2d12f489da2a93e862bb57760 (127.5.70.2:34929) I1121 10:21:13.071357 7015 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "028c77fc16a1479d859f65a3b4a3fb98" candidate_uuid: "c86302eb47f945cf8b90bc1c62a703c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd71d3e0796243de9ac2a8ca6d433918" I1121 10:21:13.071383 6643 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "028c77fc16a1479d859f65a3b4a3fb98" candidate_uuid: "c86302eb47f945cf8b90bc1c62a703c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05aeabf2d12f489da2a93e862bb57760" I1121 10:21:13.071391 7015 raft_consensus.cc:2905] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:13.071404 6643 raft_consensus.cc:2905] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:13.072089 7015 raft_consensus.cc:2359] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86302eb47f945cf8b90bc1c62a703c5 in term 1. I1121 10:21:13.072160 6643 raft_consensus.cc:2359] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86302eb47f945cf8b90bc1c62a703c5 in term 1. I1121 10:21:13.072261 6700 leader_election.cc:310] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86302eb47f945cf8b90bc1c62a703c5, fd71d3e0796243de9ac2a8ca6d433918; no voters: I1121 10:21:13.072325 7079 raft_consensus.cc:2653] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:13.072435 7079 raft_consensus.cc:664] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [term 1 LEADER]: Becoming Leader. State: Replica: c86302eb47f945cf8b90bc1c62a703c5, State: Running, Role: LEADER I1121 10:21:13.072558 7079 consensus_queue.cc:208] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } } I1121 10:21:13.073511 6395 catalog_manager.cc:4143] T 028c77fc16a1479d859f65a3b4a3fb98 P c86302eb47f945cf8b90bc1c62a703c5 reported cstate change: term changed from 0 to 1, leader changed from to c86302eb47f945cf8b90bc1c62a703c5 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "c86302eb47f945cf8b90bc1c62a703c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86302eb47f945cf8b90bc1c62a703c5" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd71d3e0796243de9ac2a8ca6d433918" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 40777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05aeabf2d12f489da2a93e862bb57760" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 34929 } health_report { overall_health: UNKNOWN } } } I1121 10:21:13.094570 5400 external_mini_cluster.cc:717] 5 TS(s) registered with all masters I1121 10:21:13.095002 5400 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6f5b7acb7f9d4a6389c4ce907c928f7e to finish bootstrapping I1121 10:21:13.098814 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 05aeabf2d12f489da2a93e862bb57760 to finish bootstrapping I1121 10:21:13.101804 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c86302eb47f945cf8b90bc1c62a703c5 to finish bootstrapping I1121 10:21:13.104202 5400 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 17a86b7e343646309c0acbd83b846e4d to finish bootstrapping I1121 10:21:13.107530 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fd71d3e0796243de9ac2a8ca6d433918 to finish bootstrapping W1121 10:21:13.184648 7059 tablet.cc:1767] T 028c77fc16a1479d859f65a3b4a3fb98 P fd71d3e0796243de9ac2a8ca6d433918: Can't schedule compaction. Clean time has not been advanced past its initial value. W1121 10:21:13.186333 6687 tablet.cc:1767] T 028c77fc16a1479d859f65a3b4a3fb98 P 05aeabf2d12f489da2a93e862bb57760: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:13.392021 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 6439 I1121 10:21:13.395155 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 6565 I1121 10:21:13.398188 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 6689 I1121 10:21:13.401458 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 6813 I1121 10:21:13.404297 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 6937 I1121 10:21:13.407178 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master with pid 6368 2019-11-21T10:21:13Z chronyd exiting [ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (1063 ms) [----------] 1 test from RebalanceStartSafetyTest (1063 ms total) [----------] 1 test from RebalanceParamTest [ RUN ] RebalanceParamTest.Rebalance/7 I1121 10:21:13.425144 5400 test_util.cc:225] Using random seed: -1473825546 I1121 10:21:13.425597 5400 ts_itest-base.cc:112] Starting cluster with: I1121 10:21:13.425613 5400 ts_itest-base.cc:113] -------------- I1121 10:21:13.425616 5400 ts_itest-base.cc:114] 7 tablet servers I1121 10:21:13.425619 5400 ts_itest-base.cc:115] 5 replicas per TS I1121 10:21:13.425622 5400 ts_itest-base.cc:116] -------------- 2019-11-21T10:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2019-11-21T10:21:13Z Disabled control of system clock I1121 10:21:13.431933 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:38839 --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --allow_unsafe_replication_factor --raft_prepare_replacement_before_eviction=true --tserver_unresponsive_timeout_ms=3000 --rpc_reuseport=true --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:13.476114 7252 flags.cc:404] Enabled unsafe flag: --allow_unsafe_replication_factor=true W1121 10:21:13.476291 7252 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:13.476306 7252 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:13.477273 7252 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:13.477298 7252 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W1121 10:21:13.477305 7252 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:13.477310 7252 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W1121 10:21:13.477316 7252 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I1121 10:21:13.478776 7252 master_main.cc:98] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --ipki_ca_key_size=1024 --tserver_unresponsive_timeout_ms=3000 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:38839 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.7252 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:13.480777 7252 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:13.481964 7252 hybrid_clock.cc:183] HybridClock initialized: now 1574331673481896 us; error 77 us; skew 500 ppm I1121 10:21:13.482128 7252 fs_manager.cc:261] Metadata directory not provided I1121 10:21:13.482168 7252 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:13.482275 7252 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:13.482296 7252 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:13.483378 7252 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/instance: uuid: "913850c84b1540c1945d797659799e2b" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.483599 7252 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "913850c84b1540c1945d797659799e2b" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.485159 7252 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:21:13.485474 7252 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:13.485615 7252 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:13.486380 7252 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:13.486431 7252 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal uuid: "913850c84b1540c1945d797659799e2b" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.486491 7252 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:13.522467 7252 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:13.525959 7252 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.62:38839 I1121 10:21:13.526310 7252 webserver.cc:382] Webserver started at http://127.5.70.62:34565/ using document root and password file I1121 10:21:13.526628 7252 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb I1121 10:21:13.527200 7313 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:13.529743 7313 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b: Bootstrap starting. I1121 10:21:13.530215 7313 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b: No blocks or log segments found. Creating new log. I1121 10:21:13.530429 7313 log.cc:810] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b: Log is configured to *not* fsync() on all Append() calls I1121 10:21:13.530947 7313 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b: No bootstrap required, opened a new log I1121 10:21:13.532232 7313 raft_consensus.cc:337] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913850c84b1540c1945d797659799e2b" member_type: VOTER } I1121 10:21:13.532272 7313 raft_consensus.cc:363] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:13.532284 7313 raft_consensus.cc:702] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913850c84b1540c1945d797659799e2b, State: Initialized, Role: FOLLOWER I1121 10:21:13.532416 7313 consensus_queue.cc:231] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913850c84b1540c1945d797659799e2b" member_type: VOTER } I1121 10:21:13.532441 7313 raft_consensus.cc:377] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:21:13.532452 7313 raft_consensus.cc:469] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:21:13.532474 7313 raft_consensus.cc:2905] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:13.533011 7313 raft_consensus.cc:491] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913850c84b1540c1945d797659799e2b" member_type: VOTER } I1121 10:21:13.533085 7313 leader_election.cc:310] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 913850c84b1540c1945d797659799e2b; no voters: I1121 10:21:13.533190 7313 leader_election.cc:296] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:21:13.533326 7316 raft_consensus.cc:2653] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:13.533329 7313 sys_catalog.cc:340] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913850c84b1540c1945d797659799e2b" member_type: VOTER } } I1121 10:21:13.533407 7313 sys_catalog.cc:343] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [sys.catalog]: This master's current role is: FOLLOWER I1121 10:21:13.533468 7316 raft_consensus.cc:664] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [term 1 LEADER]: Becoming Leader. State: Replica: 913850c84b1540c1945d797659799e2b, State: Running, Role: LEADER I1121 10:21:13.533540 7316 consensus_queue.cc:208] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913850c84b1540c1945d797659799e2b" member_type: VOTER } I1121 10:21:13.533819 7315 sys_catalog.cc:340] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "913850c84b1540c1945d797659799e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913850c84b1540c1945d797659799e2b" member_type: VOTER } } I1121 10:21:13.533857 7315 sys_catalog.cc:343] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [sys.catalog]: This master's current role is: LEADER I1121 10:21:13.533808 7313 sys_catalog.cc:428] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [sys.catalog]: configured and running, proceeding with master startup. I1121 10:21:13.533877 7317 sys_catalog.cc:340] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 913850c84b1540c1945d797659799e2b. Latest consensus state: current_term: 1 leader_uuid: "913850c84b1540c1945d797659799e2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "913850c84b1540c1945d797659799e2b" member_type: VOTER } } I1121 10:21:13.533895 7317 sys_catalog.cc:343] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b [sys.catalog]: This master's current role is: LEADER W1121 10:21:13.534862 7320 catalog_manager.cc:1159] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b: acquiring CA information for follower catalog manager: Not found: root CA entry not found W1121 10:21:13.534891 7320 catalog_manager.cc:631] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I1121 10:21:13.535656 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master as pid 7252 I1121 10:21:13.535971 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.70.1:0 --local_ip_for_outbound_sockets=127.5.70.1 --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:21:13.536558 7319 catalog_manager.cc:1090] Loading table and tablet metadata into memory... I1121 10:21:13.536727 7319 catalog_manager.cc:1101] Initializing Kudu internal certificate authority... WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:13.586625 7323 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:13.586819 7323 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:13.587927 7323 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:13.587950 7323 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:13.587968 7323 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.1 I1121 10:21:13.589620 7323 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7323 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:13.592197 7323 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:13.593366 7323 hybrid_clock.cc:183] HybridClock initialized: now 1574331673593334 us; error 37 us; skew 500 ppm I1121 10:21:13.593551 7323 fs_manager.cc:261] Metadata directory not provided I1121 10:21:13.593600 7323 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:13.593721 7323 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:13.593752 7323 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:13.598816 7323 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "63b9dc8cf49b47ebb644515af2f427e7" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.599115 7323 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "63b9dc8cf49b47ebb644515af2f427e7" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.601119 7323 fs_manager.cc:508] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s I1121 10:21:13.601464 7323 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:13.601588 7323 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:13.602452 7323 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:13.602491 7323 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal uuid: "63b9dc8cf49b47ebb644515af2f427e7" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.602543 7323 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:13.620615 7319 catalog_manager.cc:985] Generated new certificate authority record I1121 10:21:13.621301 7319 catalog_manager.cc:1110] Loading token signing keys... I1121 10:21:13.628294 7319 catalog_manager.cc:4455] T 00000000000000000000000000000000 P 913850c84b1540c1945d797659799e2b: Generated new TSK 0 I1121 10:21:13.628415 7319 catalog_manager.cc:1120] Initializing in-progress tserver states... I1121 10:21:13.650439 7323 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:13.650846 7323 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:13.650871 7323 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:13.656395 7323 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.1:36995 I1121 10:21:13.657099 7323 webserver.cc:382] Webserver started at http://127.5.70.1:35899/ using document root and password file I1121 10:21:13.657364 7323 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb I1121 10:21:13.660629 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 7323 I1121 10:21:13.661036 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.70.2:0 --local_ip_for_outbound_sockets=127.5.70.2 --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:21:13.669535 7445 heartbeater.cc:322] Connected to a master server at 127.5.70.62:38839 I1121 10:21:13.669586 7445 heartbeater.cc:406] Registering TS with master... I1121 10:21:13.669688 7445 heartbeater.cc:455] Master 127.5.70.62:38839 requested a full tablet report, sending... I1121 10:21:13.670120 7280 ts_manager.cc:192] Registered new tserver with Master: 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995) I1121 10:21:13.670950 7280 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.1:48923 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:13.731492 7449 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:13.731643 7449 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:13.732496 7449 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:13.732547 7449 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:13.732568 7449 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.2 I1121 10:21:13.733690 7449 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7449 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.2 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:13.735486 7449 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:13.736569 7449 hybrid_clock.cc:183] HybridClock initialized: now 1574331673736500 us; error 78 us; skew 500 ppm I1121 10:21:13.736714 7449 fs_manager.cc:261] Metadata directory not provided I1121 10:21:13.736745 7449 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:13.736893 7449 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:13.736914 7449 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:13.738106 7449 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "41118f5c2e8f48179ffbebbfb6da2edf" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.738317 7449 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "41118f5c2e8f48179ffbebbfb6da2edf" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.740007 7449 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:21:13.740345 7449 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:13.740468 7449 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:13.741230 7449 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:13.741271 7449 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal uuid: "41118f5c2e8f48179ffbebbfb6da2edf" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.741327 7449 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:13.775146 7449 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:13.775575 7449 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:13.775596 7449 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:13.780856 7449 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.2:39291 I1121 10:21:13.781177 7449 webserver.cc:382] Webserver started at http://127.5.70.2:45741/ using document root and password file I1121 10:21:13.781442 7449 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb I1121 10:21:13.785583 7570 heartbeater.cc:322] Connected to a master server at 127.5.70.62:38839 I1121 10:21:13.785629 7570 heartbeater.cc:406] Registering TS with master... I1121 10:21:13.785728 7570 heartbeater.cc:455] Master 127.5.70.62:38839 requested a full tablet report, sending... I1121 10:21:13.786017 7280 ts_manager.cc:192] Registered new tserver with Master: 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291) I1121 10:21:13.786582 7280 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.2:49923 I1121 10:21:13.789611 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 7449 I1121 10:21:13.789876 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.70.3:0 --local_ip_for_outbound_sockets=127.5.70.3 --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:13.827955 7573 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:13.828099 7573 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:13.828893 7573 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:13.828912 7573 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:13.828925 7573 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.3 I1121 10:21:13.830014 7573 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7573 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.3 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:13.831748 7573 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:13.832901 7573 hybrid_clock.cc:183] HybridClock initialized: now 1574331673832884 us; error 35 us; skew 500 ppm I1121 10:21:13.833041 7573 fs_manager.cc:261] Metadata directory not provided I1121 10:21:13.833073 7573 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:13.833176 7573 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:13.833195 7573 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:13.834223 7573 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "f5bc6d2f9c574b4db9396f9f900c1141" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.834385 7573 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "f5bc6d2f9c574b4db9396f9f900c1141" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.835791 7573 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:13.836021 7573 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:13.836117 7573 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:13.836818 7573 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:13.836851 7573 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal uuid: "f5bc6d2f9c574b4db9396f9f900c1141" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.836882 7573 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:13.894560 7573 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:13.894928 7573 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:13.894946 7573 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:13.899578 7573 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.3:43909 I1121 10:21:13.899897 7573 webserver.cc:382] Webserver started at http://127.5.70.3:36641/ using document root and password file I1121 10:21:13.900137 7573 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb I1121 10:21:13.904305 7694 heartbeater.cc:322] Connected to a master server at 127.5.70.62:38839 I1121 10:21:13.904362 7694 heartbeater.cc:406] Registering TS with master... I1121 10:21:13.904449 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 7573 I1121 10:21:13.904477 7694 heartbeater.cc:455] Master 127.5.70.62:38839 requested a full tablet report, sending... I1121 10:21:13.904810 7280 ts_manager.cc:192] Registered new tserver with Master: f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:13.904861 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.70.4:0 --local_ip_for_outbound_sockets=127.5.70.4 --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:21:13.905421 7280 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.3:59015 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:13.941290 7697 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:13.941431 7697 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:13.942221 7697 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:13.942246 7697 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:13.942260 7697 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.4 I1121 10:21:13.943358 7697 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7697 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.4 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:13.945169 7697 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:13.946292 7697 hybrid_clock.cc:183] HybridClock initialized: now 1574331673946254 us; error 48 us; skew 500 ppm I1121 10:21:13.946432 7697 fs_manager.cc:261] Metadata directory not provided I1121 10:21:13.946468 7697 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:13.946563 7697 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:13.946583 7697 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:13.947666 7697 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "1b3038ad6c564f438aa18383cddc769a" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.947882 7697 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "1b3038ad6c564f438aa18383cddc769a" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.949295 7697 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s I1121 10:21:13.949570 7697 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:13.949692 7697 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:13.950376 7697 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:13.950428 7697 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal uuid: "1b3038ad6c564f438aa18383cddc769a" format_stamp: "Formatted at 2019-11-21 10:21:13 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:13.950480 7697 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:13.990656 7697 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:13.991062 7697 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:13.991081 7697 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:13.995431 7697 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.4:32993 I1121 10:21:13.995748 7697 webserver.cc:382] Webserver started at http://127.5.70.4:37797/ using document root and password file I1121 10:21:13.996042 7697 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb I1121 10:21:13.998991 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 7697 I1121 10:21:13.999270 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.70.5:0 --local_ip_for_outbound_sockets=127.5.70.5 --webserver_interface=127.5.70.5 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:21:14.000972 7818 heartbeater.cc:322] Connected to a master server at 127.5.70.62:38839 I1121 10:21:14.001032 7818 heartbeater.cc:406] Registering TS with master... I1121 10:21:14.001154 7818 heartbeater.cc:455] Master 127.5.70.62:38839 requested a full tablet report, sending... I1121 10:21:14.001546 7280 ts_manager.cc:192] Registered new tserver with Master: 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993) I1121 10:21:14.002064 7280 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.4:42217 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:14.037009 7821 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:14.037147 7821 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:14.037926 7821 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:14.037946 7821 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:14.037959 7821 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.5 I1121 10:21:14.039108 7821 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.70.5 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7821 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.5 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:14.041086 7821 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:14.042229 7821 hybrid_clock.cc:183] HybridClock initialized: now 1574331674042228 us; error 58 us; skew 500 ppm I1121 10:21:14.042364 7821 fs_manager.cc:261] Metadata directory not provided I1121 10:21:14.042420 7821 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:14.042518 7821 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:14.042538 7821 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:14.043653 7821 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "81d8bacc3ca64d628a255ad416a77c3d" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:14.043871 7821 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "81d8bacc3ca64d628a255ad416a77c3d" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:14.045456 7821 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s I1121 10:21:14.045739 7821 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:14.045853 7821 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:14.046558 7821 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:14.046589 7821 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal uuid: "81d8bacc3ca64d628a255ad416a77c3d" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:14.046648 7821 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:14.093634 7821 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:14.094003 7821 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:14.094019 7821 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:14.098943 7821 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.5:36445 I1121 10:21:14.099236 7821 webserver.cc:382] Webserver started at http://127.5.70.5:45249/ using document root and password file I1121 10:21:14.099458 7821 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb I1121 10:21:14.103605 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 7821 I1121 10:21:14.103958 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.70.6:0 --local_ip_for_outbound_sockets=127.5.70.6 --webserver_interface=127.5.70.6 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:21:14.105259 7942 heartbeater.cc:322] Connected to a master server at 127.5.70.62:38839 I1121 10:21:14.105305 7942 heartbeater.cc:406] Registering TS with master... I1121 10:21:14.105412 7942 heartbeater.cc:455] Master 127.5.70.62:38839 requested a full tablet report, sending... I1121 10:21:14.105722 7280 ts_manager.cc:192] Registered new tserver with Master: 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:14.106315 7280 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.5:57663 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:14.140377 7945 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:14.140544 7945 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:14.141319 7945 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:14.141340 7945 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:14.141358 7945 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.6 I1121 10:21:14.142514 7945 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.6:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.70.6 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.7945 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.6 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:14.144255 7945 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:14.145382 7945 hybrid_clock.cc:183] HybridClock initialized: now 1574331674145351 us; error 41 us; skew 500 ppm I1121 10:21:14.145514 7945 fs_manager.cc:261] Metadata directory not provided I1121 10:21:14.145551 7945 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:14.145645 7945 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:14.145665 7945 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:14.146651 7945 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:14.146888 7945 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:14.148286 7945 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:14.148617 7945 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:14.148736 7945 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:14.149413 7945 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:14.149451 7945 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:14.149509 7945 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:14.183410 7945 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:14.183796 7945 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:14.183815 7945 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:14.188378 7945 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.6:32833 I1121 10:21:14.188732 7945 webserver.cc:382] Webserver started at http://127.5.70.6:46761/ using document root and password file I1121 10:21:14.188997 7945 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb I1121 10:21:14.193526 8066 heartbeater.cc:322] Connected to a master server at 127.5.70.62:38839 I1121 10:21:14.193572 8066 heartbeater.cc:406] Registering TS with master... I1121 10:21:14.193652 8066 heartbeater.cc:455] Master 127.5.70.62:38839 requested a full tablet report, sending... I1121 10:21:14.194003 7280 ts_manager.cc:192] Registered new tserver with Master: c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833) I1121 10:21:14.194478 7280 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.6:37295 I1121 10:21:14.198120 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 7945 I1121 10:21:14.198436 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.70.7:0 --local_ip_for_outbound_sockets=127.5.70.7 --webserver_interface=127.5.70.7 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:14.236343 8069 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:14.236506 8069 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:14.237330 8069 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:14.237352 8069 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:14.237371 8069 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.7 I1121 10:21:14.238513 8069 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.7:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.70.7 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.8069 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.7 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:14.240432 8069 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:14.241576 8069 hybrid_clock.cc:183] HybridClock initialized: now 1574331674241543 us; error 39 us; skew 500 ppm I1121 10:21:14.241722 8069 fs_manager.cc:261] Metadata directory not provided I1121 10:21:14.241777 8069 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:14.241888 8069 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:14.241910 8069 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:14.242996 8069 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "522e51b66e2c425091349ca2c9059244" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:14.243191 8069 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "522e51b66e2c425091349ca2c9059244" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:14.244657 8069 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:14.244976 8069 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:14.245100 8069 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:14.245893 8069 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:14.245936 8069 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal uuid: "522e51b66e2c425091349ca2c9059244" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:14.245996 8069 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:14.312026 8069 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:14.312407 8069 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:14.312427 8069 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:14.319006 8069 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.7:38147 I1121 10:21:14.319312 8069 webserver.cc:382] Webserver started at http://127.5.70.7:40471/ using document root and password file I1121 10:21:14.319586 8069 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb I1121 10:21:14.322849 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 8069 I1121 10:21:14.324291 8190 heartbeater.cc:322] Connected to a master server at 127.5.70.62:38839 I1121 10:21:14.324348 8190 heartbeater.cc:406] Registering TS with master... I1121 10:21:14.324455 8190 heartbeater.cc:455] Master 127.5.70.62:38839 requested a full tablet report, sending... I1121 10:21:14.324843 7280 ts_manager.cc:192] Registered new tserver with Master: 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147) I1121 10:21:14.325593 7280 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.7:54591 I1121 10:21:14.326721 5400 external_mini_cluster.cc:717] 7 TS(s) registered with all masters I1121 10:21:14.331826 7280 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53014: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I1121 10:21:14.338827 7629 tablet_service.cc:1023] Processing CreateTablet for tablet b0cf374217f64043803515e5125b6de0 (table=TestTable [id=5c657504639147f78690abf942e8f1b9]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:14.338842 8123 tablet_service.cc:1023] Processing CreateTablet for tablet b0cf374217f64043803515e5125b6de0 (table=TestTable [id=5c657504639147f78690abf942e8f1b9]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:14.339052 7629 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b0cf374217f64043803515e5125b6de0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:14.339052 8123 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b0cf374217f64043803515e5125b6de0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:14.339869 7874 tablet_service.cc:1023] Processing CreateTablet for tablet b0cf374217f64043803515e5125b6de0 (table=TestTable [id=5c657504639147f78690abf942e8f1b9]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:14.340075 7874 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b0cf374217f64043803515e5125b6de0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:14.342649 8213 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:14.342967 7503 tablet_service.cc:1023] Processing CreateTablet for tablet b0cf374217f64043803515e5125b6de0 (table=TestTable [id=5c657504639147f78690abf942e8f1b9]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:14.343142 7503 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b0cf374217f64043803515e5125b6de0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:14.343359 8213 tablet_bootstrap.cc:590] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:14.343734 8213 log.cc:810] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: Log is configured to *not* fsync() on all Append() calls I1121 10:21:14.344003 8214 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:14.344475 8214 tablet_bootstrap.cc:590] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: No blocks or log segments found. Creating new log. I1121 10:21:14.344648 8215 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:14.345103 8215 tablet_bootstrap.cc:590] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:14.345393 8215 log.cc:810] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d: Log is configured to *not* fsync() on all Append() calls I1121 10:21:14.345949 8217 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:14.345973 8215 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:14.346056 8215 ts_tablet_manager.cc:1104] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:14.346192 7755 tablet_service.cc:1023] Processing CreateTablet for tablet b0cf374217f64043803515e5125b6de0 (table=TestTable [id=5c657504639147f78690abf942e8f1b9]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:21:14.346400 7755 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b0cf374217f64043803515e5125b6de0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:14.346570 8217 tablet_bootstrap.cc:590] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:14.346938 8217 log.cc:810] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf: Log is configured to *not* fsync() on all Append() calls I1121 10:21:14.347486 8214 log.cc:810] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Log is configured to *not* fsync() on all Append() calls I1121 10:21:14.347677 8215 raft_consensus.cc:337] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.347730 8215 raft_consensus.cc:363] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:14.347748 8215 raft_consensus.cc:702] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:14.348042 8215 consensus_queue.cc:231] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.348079 8213 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:14.348124 8213 ts_tablet_manager.cc:1104] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s I1121 10:21:14.348248 7942 heartbeater.cc:447] Master 127.5.70.62:38839 was elected leader, sending a full tablet report... I1121 10:21:14.348429 8215 ts_tablet_manager.cc:1132] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s W1121 10:21:14.350082 7943 tablet.cc:1767] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:14.350108 8214 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: No bootstrap required, opened a new log I1121 10:21:14.350152 8214 ts_tablet_manager.cc:1104] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s I1121 10:21:14.350337 8213 raft_consensus.cc:337] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.350397 8213 raft_consensus.cc:363] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:14.350417 8213 raft_consensus.cc:702] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:14.350566 8213 consensus_queue.cc:231] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.350757 8213 ts_tablet_manager.cc:1132] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I1121 10:21:14.350805 7694 heartbeater.cc:447] Master 127.5.70.62:38839 was elected leader, sending a full tablet report... I1121 10:21:14.350929 8222 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:14.351526 8222 tablet_bootstrap.cc:590] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:14.351881 8222 log.cc:810] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a: Log is configured to *not* fsync() on all Append() calls I1121 10:21:14.352181 8214 raft_consensus.cc:337] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.352222 8214 raft_consensus.cc:363] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:14.352233 8214 raft_consensus.cc:702] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: FOLLOWER I1121 10:21:14.352336 8214 consensus_queue.cc:231] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.352551 8190 heartbeater.cc:447] Master 127.5.70.62:38839 was elected leader, sending a full tablet report... I1121 10:21:14.352748 8214 ts_tablet_manager.cc:1132] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I1121 10:21:14.352952 8222 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:14.353005 8222 ts_tablet_manager.cc:1104] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:14.353371 8217 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:14.353451 8217 ts_tablet_manager.cc:1104] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s I1121 10:21:14.355134 8222 raft_consensus.cc:337] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.355198 8222 raft_consensus.cc:363] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:14.355232 8222 raft_consensus.cc:702] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:14.355370 8222 consensus_queue.cc:231] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.355581 7818 heartbeater.cc:447] Master 127.5.70.62:38839 was elected leader, sending a full tablet report... I1121 10:21:14.355602 8222 ts_tablet_manager.cc:1132] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I1121 10:21:14.355584 8217 raft_consensus.cc:337] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.355628 8217 raft_consensus.cc:363] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:14.355646 8217 raft_consensus.cc:702] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:14.355792 8217 consensus_queue.cc:231] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.355986 7570 heartbeater.cc:447] Master 127.5.70.62:38839 was elected leader, sending a full tablet report... I1121 10:21:14.355998 8217 ts_tablet_manager.cc:1132] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s W1121 10:21:14.400851 7695 tablet.cc:1767] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:14.441692 8226 raft_consensus.cc:469] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:14.441871 8226 raft_consensus.cc:491] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.442101 8226 leader_election.cc:296] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:14.446401 8147 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b0cf374217f64043803515e5125b6de0" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522e51b66e2c425091349ca2c9059244" is_pre_election: true I1121 10:21:14.446481 8147 raft_consensus.cc:2359] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3038ad6c564f438aa18383cddc769a in term 0. I1121 10:21:14.446624 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b0cf374217f64043803515e5125b6de0" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:14.446709 7519 raft_consensus.cc:2359] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3038ad6c564f438aa18383cddc769a in term 0. I1121 10:21:14.446926 7708 leader_election.cc:310] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 41118f5c2e8f48179ffbebbfb6da2edf, 522e51b66e2c425091349ca2c9059244; no voters: I1121 10:21:14.447028 8226 raft_consensus.cc:2653] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:14.447058 8226 raft_consensus.cc:469] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:14.447057 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b0cf374217f64043803515e5125b6de0" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:14.447067 8226 raft_consensus.cc:2905] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:14.447124 7651 raft_consensus.cc:2359] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3038ad6c564f438aa18383cddc769a in term 0. I1121 10:21:14.447650 7896 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b0cf374217f64043803515e5125b6de0" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:14.447721 7896 raft_consensus.cc:2359] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b3038ad6c564f438aa18383cddc769a in term 0. I1121 10:21:14.447976 8226 raft_consensus.cc:491] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.448066 8226 leader_election.cc:296] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [CANDIDATE]: Term 1 election: Requested vote from peers 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:14.448199 7896 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b0cf374217f64043803515e5125b6de0" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" I1121 10:21:14.448222 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b0cf374217f64043803515e5125b6de0" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" I1121 10:21:14.448228 7896 raft_consensus.cc:2905] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:14.448248 7651 raft_consensus.cc:2905] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:14.448316 8147 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b0cf374217f64043803515e5125b6de0" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522e51b66e2c425091349ca2c9059244" I1121 10:21:14.448349 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b0cf374217f64043803515e5125b6de0" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:14.448376 8147 raft_consensus.cc:2905] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:14.448377 7519 raft_consensus.cc:2905] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:14.449160 7651 raft_consensus.cc:2359] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3038ad6c564f438aa18383cddc769a in term 1. I1121 10:21:14.449165 8147 raft_consensus.cc:2359] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3038ad6c564f438aa18383cddc769a in term 1. I1121 10:21:14.449165 7896 raft_consensus.cc:2359] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3038ad6c564f438aa18383cddc769a in term 1. I1121 10:21:14.449308 7519 raft_consensus.cc:2359] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3038ad6c564f438aa18383cddc769a in term 1. I1121 10:21:14.449340 7708 leader_election.cc:310] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 522e51b66e2c425091349ca2c9059244, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:14.449424 8226 raft_consensus.cc:2653] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:14.449540 8226 raft_consensus.cc:664] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 1 LEADER]: Becoming Leader. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Running, Role: LEADER I1121 10:21:14.449643 8226 consensus_queue.cc:208] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:14.450610 7279 catalog_manager.cc:4143] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a reported cstate change: term changed from 0 to 1, leader changed from to 1b3038ad6c564f438aa18383cddc769a (127.5.70.4). New cstate: current_term: 1 leader_uuid: "1b3038ad6c564f438aa18383cddc769a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } } I1121 10:21:14.454519 5400 external_mini_cluster.cc:717] 7 TS(s) registered with all masters I1121 10:21:14.454972 5400 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 63b9dc8cf49b47ebb644515af2f427e7 to finish bootstrapping I1121 10:21:14.458107 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 41118f5c2e8f48179ffbebbfb6da2edf to finish bootstrapping I1121 10:21:14.460609 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f5bc6d2f9c574b4db9396f9f900c1141 to finish bootstrapping I1121 10:21:14.463127 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1b3038ad6c564f438aa18383cddc769a to finish bootstrapping I1121 10:21:14.465524 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 81d8bacc3ca64d628a255ad416a77c3d to finish bootstrapping I1121 10:21:14.468099 5400 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c9e9c35ff1cb4b68ba7cf5c037a98347 to finish bootstrapping I1121 10:21:14.470957 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 522e51b66e2c425091349ca2c9059244 to finish bootstrapping I1121 10:21:14.473309 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 8069 W1121 10:21:14.497292 7819 tablet.cc:1767] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a: Can't schedule compaction. Clean time has not been advanced past its initial value. W1121 10:21:14.532344 7571 tablet.cc:1767] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:14.671649 7445 heartbeater.cc:447] Master 127.5.70.62:38839 was elected leader, sending a full tablet report... W1121 10:21:14.838241 7708 consensus_peers.cc:473] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Status: Network error: Client connection negotiation failed: client connection to 127.5.70.7:38147: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry. I1121 10:21:14.846876 8226 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:14.976208 8226 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:14.978176 8240 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:15.195127 8066 heartbeater.cc:447] Master 127.5.70.62:38839 was elected leader, sending a full tablet report... W1121 10:21:17.353471 7708 consensus_peers.cc:473] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Status: Network error: Client connection negotiation failed: client connection to 127.5.70.7:38147: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry. I1121 10:21:18.227820 7280 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53014: name: "rebalance_test_table_0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:18.229992 7629 tablet_service.cc:1023] Processing CreateTablet for tablet d84faff8cd0f433d8de8664fd0054d2d (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.230123 7629 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet d84faff8cd0f433d8de8664fd0054d2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.230278 7507 tablet_service.cc:1023] Processing CreateTablet for tablet 133fea61ffe5433785a3fe3736a2a668 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.230373 7507 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 133fea61ffe5433785a3fe3736a2a668. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.230389 7501 tablet_service.cc:1023] Processing CreateTablet for tablet 605a58ce874c4170b0d24d922f017068 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.230556 7501 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 605a58ce874c4170b0d24d922f017068. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.230727 7874 tablet_service.cc:1023] Processing CreateTablet for tablet 133fea61ffe5433785a3fe3736a2a668 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.230823 7874 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 133fea61ffe5433785a3fe3736a2a668. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.231110 7872 tablet_service.cc:1023] Processing CreateTablet for tablet 605a58ce874c4170b0d24d922f017068 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.231189 7872 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 605a58ce874c4170b0d24d922f017068. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.231516 7503 tablet_service.cc:1023] Processing CreateTablet for tablet d84faff8cd0f433d8de8664fd0054d2d (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.231592 7503 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet d84faff8cd0f433d8de8664fd0054d2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.231799 8258 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:18.232151 8258 tablet_bootstrap.cc:590] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:18.232404 7755 tablet_service.cc:1023] Processing CreateTablet for tablet d84faff8cd0f433d8de8664fd0054d2d (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.232451 8259 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:18.232492 7755 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet d84faff8cd0f433d8de8664fd0054d2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.232605 7754 tablet_service.cc:1023] Processing CreateTablet for tablet 605a58ce874c4170b0d24d922f017068 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.232661 7754 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 605a58ce874c4170b0d24d922f017068. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.232854 8259 tablet_bootstrap.cc:590] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:18.233238 8260 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:18.233253 7629 tablet_service.cc:1023] Processing CreateTablet for tablet 133fea61ffe5433785a3fe3736a2a668 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.233332 7629 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 133fea61ffe5433785a3fe3736a2a668. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.233577 8260 tablet_bootstrap.cc:590] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:18.236125 8259 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:18.236210 8260 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:18.236213 8259 ts_tablet_manager.cc:1104] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s I1121 10:21:18.236304 8260 ts_tablet_manager.cc:1104] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:18.236505 8260 raft_consensus.cc:337] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.236505 8259 raft_consensus.cc:337] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.236554 8259 raft_consensus.cc:363] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.236563 8259 raft_consensus.cc:702] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:18.236641 8260 raft_consensus.cc:363] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.236656 8259 consensus_queue.cc:231] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.236708 8260 raft_consensus.cc:702] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:18.236855 8260 consensus_queue.cc:231] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.237159 8259 ts_tablet_manager.cc:1132] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.237252 8259 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:18.237550 8260 ts_tablet_manager.cc:1132] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:18.237653 8260 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:18.237694 8259 tablet_bootstrap.cc:590] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:18.237761 8264 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:18.238170 8260 tablet_bootstrap.cc:590] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:18.238554 8259 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:18.238663 8259 ts_tablet_manager.cc:1104] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.238961 8259 raft_consensus.cc:337] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.239073 8259 raft_consensus.cc:363] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.239068 7372 tablet_service.cc:1023] Processing CreateTablet for tablet d84faff8cd0f433d8de8664fd0054d2d (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.239081 7371 tablet_service.cc:1023] Processing CreateTablet for tablet 133fea61ffe5433785a3fe3736a2a668 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.239140 8259 raft_consensus.cc:702] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:18.239287 8259 consensus_queue.cc:231] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.239307 7371 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 133fea61ffe5433785a3fe3736a2a668. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.239396 7372 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet d84faff8cd0f433d8de8664fd0054d2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.239424 8003 tablet_service.cc:1023] Processing CreateTablet for tablet d84faff8cd0f433d8de8664fd0054d2d (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.239589 8003 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet d84faff8cd0f433d8de8664fd0054d2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.240219 8259 ts_tablet_manager.cc:1132] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.240264 8259 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:18.240617 7370 tablet_service.cc:1023] Processing CreateTablet for tablet 605a58ce874c4170b0d24d922f017068 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.240617 8002 tablet_service.cc:1023] Processing CreateTablet for tablet 133fea61ffe5433785a3fe3736a2a668 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.240622 8259 tablet_bootstrap.cc:590] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:18.240710 8002 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 133fea61ffe5433785a3fe3736a2a668. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.240710 7370 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 605a58ce874c4170b0d24d922f017068. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.240901 8001 tablet_service.cc:1023] Processing CreateTablet for tablet 605a58ce874c4170b0d24d922f017068 (table=rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.241004 8001 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 605a58ce874c4170b0d24d922f017068. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.241586 8264 tablet_bootstrap.cc:590] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:18.242043 8259 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:18.242077 8259 ts_tablet_manager.cc:1104] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:18.242223 8259 raft_consensus.cc:337] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.242235 8259 raft_consensus.cc:363] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.242242 8259 raft_consensus.cc:702] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:18.242588 8259 consensus_queue.cc:231] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.242880 8260 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:18.242915 8260 ts_tablet_manager.cc:1104] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s I1121 10:21:18.243095 8260 raft_consensus.cc:337] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.243114 8260 raft_consensus.cc:363] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.243122 8260 raft_consensus.cc:702] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:18.243288 8259 ts_tablet_manager.cc:1132] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:18.243378 8260 consensus_queue.cc:231] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.243752 8260 ts_tablet_manager.cc:1132] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:18.244104 8258 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:18.244158 8258 ts_tablet_manager.cc:1104] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s I1121 10:21:18.244369 8258 raft_consensus.cc:337] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.244392 8258 raft_consensus.cc:363] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.244398 8258 raft_consensus.cc:702] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:18.244462 8258 consensus_queue.cc:231] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.244597 8258 ts_tablet_manager.cc:1132] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s I1121 10:21:18.244649 8258 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:18.245023 8258 tablet_bootstrap.cc:590] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:18.245674 8273 raft_consensus.cc:469] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.245782 8273 raft_consensus.cc:491] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.245999 8273 leader_election.cc:296] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291) I1121 10:21:18.246830 8272 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:18.247220 8272 tablet_bootstrap.cc:590] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:18.247440 8272 log.cc:810] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: Log is configured to *not* fsync() on all Append() calls I1121 10:21:18.247656 8275 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:18.247967 8272 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:18.248023 8272 ts_tablet_manager.cc:1104] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.248051 8275 tablet_bootstrap.cc:590] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:18.248242 8275 log.cc:810] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347: Log is configured to *not* fsync() on all Append() calls I1121 10:21:18.249733 8264 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:18.249788 8264 ts_tablet_manager.cc:1104] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s I1121 10:21:18.249951 8264 raft_consensus.cc:337] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.249977 8264 raft_consensus.cc:363] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.249985 8264 raft_consensus.cc:702] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:18.250056 8264 consensus_queue.cc:231] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.250142 8264 ts_tablet_manager.cc:1132] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.250200 8264 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:18.250505 8264 tablet_bootstrap.cc:590] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:18.251035 8258 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:18.251057 7390 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "d84faff8cd0f433d8de8664fd0054d2d" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" is_pre_election: true I1121 10:21:18.251089 8258 ts_tablet_manager.cc:1104] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s I1121 10:21:18.251291 8258 raft_consensus.cc:337] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.251317 8258 raft_consensus.cc:363] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.251324 8258 raft_consensus.cc:702] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:18.251400 8258 consensus_queue.cc:231] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.251469 8258 ts_tablet_manager.cc:1132] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s W1121 10:21:18.251638 7583 leader_election.cc:348] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995): Illegal state: must be running to vote when last-logged opid is not known I1121 10:21:18.252766 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "d84faff8cd0f433d8de8664fd0054d2d" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true W1121 10:21:18.252990 7583 leader_election.cc:348] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833): Illegal state: must be running to vote when last-logged opid is not known I1121 10:21:18.253504 8264 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:18.253542 8264 ts_tablet_manager.cc:1104] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:18.253684 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "d84faff8cd0f433d8de8664fd0054d2d" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:18.253712 8264 raft_consensus.cc:337] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.253720 7519 raft_consensus.cc:2359] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.253727 8264 raft_consensus.cc:363] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.253736 8264 raft_consensus.cc:702] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:18.253808 8264 consensus_queue.cc:231] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.253917 8264 ts_tablet_manager.cc:1132] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:18.254267 8275 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:18.254380 8275 ts_tablet_manager.cc:1104] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s I1121 10:21:18.254475 8272 raft_consensus.cc:337] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.254544 8272 raft_consensus.cc:363] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.254568 8272 raft_consensus.cc:702] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:18.254763 8272 consensus_queue.cc:231] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.254941 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "d84faff8cd0f433d8de8664fd0054d2d" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:18.254990 7775 raft_consensus.cc:2359] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.255069 8272 ts_tablet_manager.cc:1132] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s I1121 10:21:18.255138 8272 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:18.255318 7584 leader_election.cc:310] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 41118f5c2e8f48179ffbebbfb6da2edf, f5bc6d2f9c574b4db9396f9f900c1141; no voters: 63b9dc8cf49b47ebb644515af2f427e7, c9e9c35ff1cb4b68ba7cf5c037a98347 I1121 10:21:18.255520 8273 raft_consensus.cc:2653] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:18.255544 8272 tablet_bootstrap.cc:590] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:18.255558 8273 raft_consensus.cc:469] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:18.255570 8273 raft_consensus.cc:2905] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.255977 8275 raft_consensus.cc:337] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.256037 8275 raft_consensus.cc:363] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.256057 8275 raft_consensus.cc:702] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:18.256119 8272 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:18.256152 8272 ts_tablet_manager.cc:1104] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.256217 8275 consensus_queue.cc:231] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.256373 8273 raft_consensus.cc:491] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.256381 8272 raft_consensus.cc:337] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.256403 8272 raft_consensus.cc:363] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.256410 8272 raft_consensus.cc:702] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:18.256599 8275 ts_tablet_manager.cc:1132] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:18.256618 8272 consensus_queue.cc:231] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.256628 7390 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "d84faff8cd0f433d8de8664fd0054d2d" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" I1121 10:21:18.256659 7390 raft_consensus.cc:2905] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.256672 8275 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:18.256692 8272 ts_tablet_manager.cc:1132] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.256707 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "d84faff8cd0f433d8de8664fd0054d2d" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:18.256736 8272 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:18.256738 8023 raft_consensus.cc:2905] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.257017 8275 tablet_bootstrap.cc:590] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:18.257128 8272 tablet_bootstrap.cc:590] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:18.257416 8023 raft_consensus.cc:2359] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.257426 7390 raft_consensus.cc:2359] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.257501 8273 leader_election.cc:296] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 election: Requested vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291) I1121 10:21:18.257632 7583 leader_election.cc:310] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, c9e9c35ff1cb4b68ba7cf5c037a98347, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:18.257694 8273 raft_consensus.cc:2653] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:18.257762 8275 tablet_bootstrap.cc:439] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:18.257789 8273 raft_consensus.cc:664] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Becoming Leader. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Running, Role: LEADER I1121 10:21:18.257797 8275 ts_tablet_manager.cc:1104] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.257817 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "d84faff8cd0f433d8de8664fd0054d2d" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:18.257840 7519 raft_consensus.cc:2905] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.257886 8273 consensus_queue.cc:208] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.257977 8275 raft_consensus.cc:337] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.257990 8275 raft_consensus.cc:363] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.258008 8275 raft_consensus.cc:702] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:18.258082 8275 consensus_queue.cc:231] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.258141 8275 ts_tablet_manager.cc:1132] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.258173 8275 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:18.258404 7519 raft_consensus.cc:2359] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.258424 8275 tablet_bootstrap.cc:590] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:18.258576 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "d84faff8cd0f433d8de8664fd0054d2d" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:18.258601 7775 raft_consensus.cc:2905] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.258999 8272 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:18.259035 7280 catalog_manager.cc:4143] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: term changed from 0 to 1, leader changed from to f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } } I1121 10:21:18.259043 8272 ts_tablet_manager.cc:1104] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:18.259213 8272 raft_consensus.cc:337] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.259227 8272 raft_consensus.cc:363] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.259233 8272 raft_consensus.cc:702] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:18.259315 8272 consensus_queue.cc:231] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.259332 7775 raft_consensus.cc:2359] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.259374 8272 ts_tablet_manager.cc:1132] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.260357 8275 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:18.260401 8275 ts_tablet_manager.cc:1104] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:18.260613 8275 raft_consensus.cc:337] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.260633 8275 raft_consensus.cc:363] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.260640 8275 raft_consensus.cc:702] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:18.260717 8275 consensus_queue.cc:231] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.260861 8275 ts_tablet_manager.cc:1132] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s W1121 10:21:18.413493 7446 tablet.cc:1767] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:18.422519 8266 raft_consensus.cc:469] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.422618 8266 raft_consensus.cc:491] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.422821 8266 leader_election.cc:296] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291) I1121 10:21:18.426467 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "605a58ce874c4170b0d24d922f017068" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:18.426517 7519 raft_consensus.cc:2359] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:18.426873 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "605a58ce874c4170b0d24d922f017068" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:18.426913 8023 raft_consensus.cc:2359] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:18.427069 7831 leader_election.cc:310] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, 81d8bacc3ca64d628a255ad416a77c3d, c9e9c35ff1cb4b68ba7cf5c037a98347; no voters: I1121 10:21:18.427171 8266 raft_consensus.cc:2653] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:18.427202 8266 raft_consensus.cc:469] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:18.427211 8266 raft_consensus.cc:2905] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.427624 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "605a58ce874c4170b0d24d922f017068" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:18.427667 7775 raft_consensus.cc:2359] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:18.427726 7390 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "605a58ce874c4170b0d24d922f017068" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" is_pre_election: true I1121 10:21:18.427767 7390 raft_consensus.cc:2359] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:18.428099 8266 raft_consensus.cc:491] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.428189 8266 leader_election.cc:296] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 election: Requested vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291) I1121 10:21:18.428344 7390 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "605a58ce874c4170b0d24d922f017068" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" I1121 10:21:18.428359 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "605a58ce874c4170b0d24d922f017068" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:18.428369 7390 raft_consensus.cc:2905] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.428377 8023 raft_consensus.cc:2905] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.428405 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "605a58ce874c4170b0d24d922f017068" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:18.428427 7519 raft_consensus.cc:2905] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.428444 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "605a58ce874c4170b0d24d922f017068" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:18.428462 7775 raft_consensus.cc:2905] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.429091 8023 raft_consensus.cc:2359] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:18.429124 7519 raft_consensus.cc:2359] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:18.429124 7775 raft_consensus.cc:2359] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:18.429200 7390 raft_consensus.cc:2359] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:18.429309 7831 leader_election.cc:310] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d, c9e9c35ff1cb4b68ba7cf5c037a98347; no voters: I1121 10:21:18.429376 8266 raft_consensus.cc:2653] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:18.429527 8266 raft_consensus.cc:664] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Becoming Leader. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Running, Role: LEADER I1121 10:21:18.429613 8266 consensus_queue.cc:208] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:18.430454 7278 catalog_manager.cc:4143] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: term changed from 0 to 1, leader changed from to 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5). New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } } I1121 10:21:18.436391 8286 raft_consensus.cc:469] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.436486 8286 raft_consensus.cc:491] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.436741 8286 leader_election.cc:296] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:18.440629 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "133fea61ffe5433785a3fe3736a2a668" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:18.440673 7519 raft_consensus.cc:2359] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.440798 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "133fea61ffe5433785a3fe3736a2a668" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:18.440830 8023 raft_consensus.cc:2359] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.440863 7896 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "133fea61ffe5433785a3fe3736a2a668" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:18.440897 7896 raft_consensus.cc:2359] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.441001 7333 leader_election.cc:310] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, 63b9dc8cf49b47ebb644515af2f427e7, c9e9c35ff1cb4b68ba7cf5c037a98347; no voters: I1121 10:21:18.441113 8286 raft_consensus.cc:2653] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:18.441138 8286 raft_consensus.cc:469] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:18.441146 8286 raft_consensus.cc:2905] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.441334 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "133fea61ffe5433785a3fe3736a2a668" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:18.441375 7651 raft_consensus.cc:2359] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. W1121 10:21:18.441622 8067 tablet.cc:1767] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:18.441946 8286 raft_consensus.cc:491] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.442116 8286 leader_election.cc:296] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Requested vote from peers c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:18.442196 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "133fea61ffe5433785a3fe3736a2a668" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:18.442219 7896 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "133fea61ffe5433785a3fe3736a2a668" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" I1121 10:21:18.442230 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "133fea61ffe5433785a3fe3736a2a668" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:18.442237 7896 raft_consensus.cc:2905] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.442237 8023 raft_consensus.cc:2905] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.442251 7519 raft_consensus.cc:2905] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.442822 7519 raft_consensus.cc:2359] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.442903 7896 raft_consensus.cc:2359] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.442903 8023 raft_consensus.cc:2359] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.443063 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "133fea61ffe5433785a3fe3736a2a668" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" I1121 10:21:18.443073 7334 leader_election.cc:310] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d; no voters: I1121 10:21:18.443253 8286 raft_consensus.cc:2653] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:18.443259 7651 raft_consensus.cc:2905] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.443378 8286 raft_consensus.cc:664] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Becoming Leader. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Running, Role: LEADER I1121 10:21:18.443486 8286 consensus_queue.cc:208] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:18.443897 7651 raft_consensus.cc:2359] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.444290 7278 catalog_manager.cc:4143] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: term changed from 0 to 1, leader changed from to 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } } W1121 10:21:18.523041 8309 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_0' I1121 10:21:18.555296 7519 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.555514 7775 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.555554 8023 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.555712 7390 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.555867 8299 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.556020 8266 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.556218 8266 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.559729 8327 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.561007 8325 mvcc.cc:203] Tried to move safe_time back from 6448462555360202752 to 6448462554848141312. Current Snapshot: MvccSnapshot[committed={T|T < 6448462555360202752}] I1121 10:21:18.561748 7390 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.562129 8291 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.562618 7775 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.562783 7527 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.562932 8329 mvcc.cc:203] Tried to move safe_time back from 6448462555360202752 to 6448462554848141312. Current Snapshot: MvccSnapshot[committed={T|T < 6448462555360202752}] I1121 10:21:18.563318 8273 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.563383 8022 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.563441 8291 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.564205 8273 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.564801 8331 mvcc.cc:203] Tried to move safe_time back from 6448462555368452096 to 6448462554848141312. Current Snapshot: MvccSnapshot[committed={T|T < 6448462555360202752}] I1121 10:21:18.566740 8334 mvcc.cc:203] Tried to move safe_time back from 6448462555360202752 to 6448462554848141312. Current Snapshot: MvccSnapshot[committed={T|T < 6448462555360202752}] I1121 10:21:18.567464 8346 mvcc.cc:203] Tried to move safe_time back from 6448462555360202752 to 6448462554848141312. Current Snapshot: MvccSnapshot[committed={T|T < 6448462555360202752}] I1121 10:21:18.571504 8333 mvcc.cc:203] Tried to move safe_time back from 6448462555392638976 to 6448462554144649216. Current Snapshot: MvccSnapshot[committed={T|T < 6448462555384758272}] I1121 10:21:18.575443 7651 raft_consensus.cc:1181] T 133fea61ffe5433785a3fe3736a2a668 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.575546 7896 raft_consensus.cc:1181] T 133fea61ffe5433785a3fe3736a2a668 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.575814 8286 consensus_queue.cc:984] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.575927 8286 consensus_queue.cc:984] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.576692 8022 raft_consensus.cc:1181] T 133fea61ffe5433785a3fe3736a2a668 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.576941 8307 consensus_queue.cc:984] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.578413 7527 raft_consensus.cc:1181] T 133fea61ffe5433785a3fe3736a2a668 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.578729 8286 consensus_queue.cc:984] T 133fea61ffe5433785a3fe3736a2a668 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 43.3911 ms time per row: 0.0216955 ms I1121 10:21:18.586549 7278 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53014: name: "rebalance_test_table_1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:18.588017 7627 tablet_service.cc:1023] Processing CreateTablet for tablet 787130b10eda4a71b2212bdb7ed2f140 (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.588045 7629 tablet_service.cc:1023] Processing CreateTablet for tablet 7d59736e513f426095bedf84bbeec48d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.588063 7370 tablet_service.cc:1023] Processing CreateTablet for tablet 7d59736e513f426095bedf84bbeec48d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.588133 7627 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 787130b10eda4a71b2212bdb7ed2f140. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.588156 7370 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7d59736e513f426095bedf84bbeec48d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.588182 7628 tablet_service.cc:1023] Processing CreateTablet for tablet c5abf7a6c891418482e200e7115e2c2d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.588284 8001 tablet_service.cc:1023] Processing CreateTablet for tablet 7d59736e513f426095bedf84bbeec48d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.588367 8001 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7d59736e513f426095bedf84bbeec48d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.588373 7503 tablet_service.cc:1023] Processing CreateTablet for tablet 787130b10eda4a71b2212bdb7ed2f140 (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.588456 7503 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 787130b10eda4a71b2212bdb7ed2f140. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.588591 7628 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c5abf7a6c891418482e200e7115e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.588594 8002 tablet_service.cc:1023] Processing CreateTablet for tablet c5abf7a6c891418482e200e7115e2c2d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.588611 7371 tablet_service.cc:1023] Processing CreateTablet for tablet c5abf7a6c891418482e200e7115e2c2d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.588656 8002 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c5abf7a6c891418482e200e7115e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.588706 7874 tablet_service.cc:1023] Processing CreateTablet for tablet 7d59736e513f426095bedf84bbeec48d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.588790 7874 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7d59736e513f426095bedf84bbeec48d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.588929 7872 tablet_service.cc:1023] Processing CreateTablet for tablet c5abf7a6c891418482e200e7115e2c2d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.589037 7372 tablet_service.cc:1023] Processing CreateTablet for tablet 787130b10eda4a71b2212bdb7ed2f140 (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.589303 7754 tablet_service.cc:1023] Processing CreateTablet for tablet c5abf7a6c891418482e200e7115e2c2d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.589382 7754 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c5abf7a6c891418482e200e7115e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.589534 7753 tablet_service.cc:1023] Processing CreateTablet for tablet 787130b10eda4a71b2212bdb7ed2f140 (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.589649 7871 tablet_service.cc:1023] Processing CreateTablet for tablet 787130b10eda4a71b2212bdb7ed2f140 (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.589694 7755 tablet_service.cc:1023] Processing CreateTablet for tablet 7d59736e513f426095bedf84bbeec48d (table=rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.589763 7371 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c5abf7a6c891418482e200e7115e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.589931 7629 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7d59736e513f426095bedf84bbeec48d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.590082 7372 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 787130b10eda4a71b2212bdb7ed2f140. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.591871 7753 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 787130b10eda4a71b2212bdb7ed2f140. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.592012 7755 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7d59736e513f426095bedf84bbeec48d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.592170 8258 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:18.592702 8258 tablet_bootstrap.cc:590] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:18.592865 8259 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:18.593158 7872 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c5abf7a6c891418482e200e7115e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.593212 8259 tablet_bootstrap.cc:590] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:18.593232 8264 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:18.593255 7871 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 787130b10eda4a71b2212bdb7ed2f140. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.593551 8264 tablet_bootstrap.cc:590] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:18.594132 8259 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:18.594167 8259 ts_tablet_manager.cc:1104] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:18.594274 8264 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:18.594295 8260 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:18.594323 8264 ts_tablet_manager.cc:1104] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.594379 8259 raft_consensus.cc:337] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.594395 8259 raft_consensus.cc:363] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.594403 8259 raft_consensus.cc:702] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:18.594466 8259 consensus_queue.cc:231] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.594465 8264 raft_consensus.cc:337] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.594480 8264 raft_consensus.cc:363] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.594486 8264 raft_consensus.cc:702] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:18.594535 8259 ts_tablet_manager.cc:1132] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.594550 8264 consensus_queue.cc:231] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.594611 8264 ts_tablet_manager.cc:1132] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.594643 8260 tablet_bootstrap.cc:590] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:18.594916 8258 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:18.594943 8258 ts_tablet_manager.cc:1104] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I1121 10:21:18.595085 8258 raft_consensus.cc:337] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.595096 8258 raft_consensus.cc:363] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.595100 8258 raft_consensus.cc:702] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:18.595152 8258 consensus_queue.cc:231] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.595219 8258 ts_tablet_manager.cc:1132] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.595250 8258 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:18.595541 8258 tablet_bootstrap.cc:590] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:18.595782 8272 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:18.596105 8272 tablet_bootstrap.cc:590] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:18.596226 8258 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:18.596256 8258 ts_tablet_manager.cc:1104] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:18.596392 8258 raft_consensus.cc:337] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.596405 8258 raft_consensus.cc:363] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.596410 8258 raft_consensus.cc:702] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:18.596472 8258 consensus_queue.cc:231] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.596496 8264 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:18.596649 8258 ts_tablet_manager.cc:1132] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.596684 8258 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:18.596922 8264 tablet_bootstrap.cc:590] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:18.597012 8258 tablet_bootstrap.cc:590] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:18.597484 8272 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:18.597513 8272 ts_tablet_manager.cc:1104] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:18.597683 8272 raft_consensus.cc:337] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.597699 8272 raft_consensus.cc:363] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.597707 8272 raft_consensus.cc:702] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:18.597779 8272 consensus_queue.cc:231] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.597854 8272 ts_tablet_manager.cc:1132] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.597873 8260 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:18.597901 8260 ts_tablet_manager.cc:1104] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s I1121 10:21:18.598057 8260 raft_consensus.cc:337] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.598079 8260 raft_consensus.cc:363] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.598096 8260 raft_consensus.cc:702] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:18.598157 8260 consensus_queue.cc:231] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.598170 8272 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:18.598235 8260 ts_tablet_manager.cc:1132] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.598500 8272 tablet_bootstrap.cc:590] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:18.598583 8258 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:18.598613 8258 ts_tablet_manager.cc:1104] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:18.598767 8258 raft_consensus.cc:337] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.598790 8258 raft_consensus.cc:363] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.598798 8258 raft_consensus.cc:702] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:18.598866 8258 consensus_queue.cc:231] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.598920 8258 ts_tablet_manager.cc:1132] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.599292 8272 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:18.599294 8264 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:18.599331 8264 ts_tablet_manager.cc:1104] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:18.599334 8272 ts_tablet_manager.cc:1104] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.599458 8264 raft_consensus.cc:337] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.599464 8272 raft_consensus.cc:337] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.599481 8272 raft_consensus.cc:363] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.599481 8264 raft_consensus.cc:363] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.599488 8272 raft_consensus.cc:702] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:18.599501 8264 raft_consensus.cc:702] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:18.599553 8272 consensus_queue.cc:231] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.599567 8264 consensus_queue.cc:231] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.599609 8272 ts_tablet_manager.cc:1132] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.599627 8264 ts_tablet_manager.cc:1132] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.599639 8272 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:18.599659 8264 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:18.599947 8272 tablet_bootstrap.cc:590] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:18.599990 8264 tablet_bootstrap.cc:590] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:18.600432 8260 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:18.600625 8264 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:18.600661 8264 ts_tablet_manager.cc:1104] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.600811 8264 raft_consensus.cc:337] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.600824 8264 raft_consensus.cc:363] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.600831 8264 raft_consensus.cc:702] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:18.600899 8264 consensus_queue.cc:231] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.600924 8260 tablet_bootstrap.cc:590] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:18.600961 8264 ts_tablet_manager.cc:1132] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.601233 8272 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:18.601266 8272 ts_tablet_manager.cc:1104] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:18.601408 8272 raft_consensus.cc:337] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.601420 8272 raft_consensus.cc:363] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.601426 8272 raft_consensus.cc:702] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:18.601508 8272 consensus_queue.cc:231] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.601558 8272 ts_tablet_manager.cc:1132] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.602061 8260 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:18.602097 8260 ts_tablet_manager.cc:1104] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:18.602180 8275 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:18.602246 8260 raft_consensus.cc:337] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.602258 8260 raft_consensus.cc:363] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.602265 8260 raft_consensus.cc:702] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:18.602325 8260 consensus_queue.cc:231] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.602404 8260 ts_tablet_manager.cc:1132] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.602438 8260 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:18.602501 8275 tablet_bootstrap.cc:590] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:18.602789 8260 tablet_bootstrap.cc:590] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:18.603490 8275 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:18.603533 8275 ts_tablet_manager.cc:1104] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.603678 8275 raft_consensus.cc:337] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.603696 8275 raft_consensus.cc:363] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.603703 8260 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:18.603709 8275 raft_consensus.cc:702] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:18.603730 8260 ts_tablet_manager.cc:1104] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.603775 8275 consensus_queue.cc:231] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.603847 8275 ts_tablet_manager.cc:1132] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.603886 8275 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:18.603912 8260 raft_consensus.cc:337] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.603936 8260 raft_consensus.cc:363] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.603945 8260 raft_consensus.cc:702] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:18.604005 8260 consensus_queue.cc:231] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.604074 8260 ts_tablet_manager.cc:1132] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:18.604246 8275 tablet_bootstrap.cc:590] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:18.605115 8275 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:18.605149 8275 ts_tablet_manager.cc:1104] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:18.605301 8275 raft_consensus.cc:337] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.605325 8275 raft_consensus.cc:363] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.605334 8275 raft_consensus.cc:702] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:18.605382 8291 raft_consensus.cc:469] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.605397 8275 consensus_queue.cc:231] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.605479 8275 ts_tablet_manager.cc:1132] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.605479 8291 raft_consensus.cc:491] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.605612 8291 leader_election.cc:296] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:18.605816 7390 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "787130b10eda4a71b2212bdb7ed2f140" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" is_pre_election: true I1121 10:21:18.605850 7390 raft_consensus.cc:2359] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.605942 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "787130b10eda4a71b2212bdb7ed2f140" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:18.605970 7775 raft_consensus.cc:2359] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.606133 7527 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "787130b10eda4a71b2212bdb7ed2f140" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:18.606168 7584 leader_election.cc:310] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 63b9dc8cf49b47ebb644515af2f427e7, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:18.606179 7527 raft_consensus.cc:2359] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.606222 8291 raft_consensus.cc:2653] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:18.606245 8291 raft_consensus.cc:469] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:18.606252 8291 raft_consensus.cc:2905] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.606938 8291 raft_consensus.cc:491] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.607017 8291 leader_election.cc:296] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 election: Requested vote from peers 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:18.607197 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "787130b10eda4a71b2212bdb7ed2f140" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:18.607218 7775 raft_consensus.cc:2905] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.607820 7775 raft_consensus.cc:2359] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.607918 7390 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "787130b10eda4a71b2212bdb7ed2f140" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" I1121 10:21:18.607960 7390 raft_consensus.cc:2905] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.608438 7390 raft_consensus.cc:2359] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.608569 7896 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "787130b10eda4a71b2212bdb7ed2f140" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:18.608594 7896 raft_consensus.cc:2359] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.608618 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "787130b10eda4a71b2212bdb7ed2f140" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" I1121 10:21:18.608639 7893 raft_consensus.cc:2905] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.608664 7527 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "787130b10eda4a71b2212bdb7ed2f140" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:18.608690 7527 raft_consensus.cc:2905] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.609266 7893 raft_consensus.cc:2359] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.609386 7527 raft_consensus.cc:2359] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.609433 7584 leader_election.cc:310] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 81d8bacc3ca64d628a255ad416a77c3d, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:18.609525 8291 raft_consensus.cc:2653] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:18.609556 8291 raft_consensus.cc:664] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Becoming Leader. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Running, Role: LEADER I1121 10:21:18.609628 8291 consensus_queue.cc:208] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.610283 7277 catalog_manager.cc:4143] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: term changed from 0 to 1, leader changed from to f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } } I1121 10:21:18.624738 8363 raft_consensus.cc:469] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.624842 8363 raft_consensus.cc:491] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.625174 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:18.625176 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:18.625214 7651 raft_consensus.cc:2359] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.625216 8022 raft_consensus.cc:2359] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.625408 8363 leader_election.cc:296] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833) I1121 10:21:18.625430 7333 leader_election.cc:310] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, c9e9c35ff1cb4b68ba7cf5c037a98347, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:18.625479 8363 raft_consensus.cc:2653] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:18.625469 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:18.625517 7893 raft_consensus.cc:2359] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.625496 8363 raft_consensus.cc:469] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:18.625563 8363 raft_consensus.cc:2905] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.626399 8363 raft_consensus.cc:491] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.626655 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" I1121 10:21:18.626683 7893 raft_consensus.cc:2905] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.626838 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" I1121 10:21:18.626897 7651 raft_consensus.cc:2905] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.627384 7893 raft_consensus.cc:2359] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.627447 7651 raft_consensus.cc:2359] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.627602 7332 leader_election.cc:310] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:18.627678 8364 raft_consensus.cc:2653] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:18.627779 8364 raft_consensus.cc:664] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Becoming Leader. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Running, Role: LEADER I1121 10:21:18.627878 8364 consensus_queue.cc:208] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.628037 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:18.628072 8022 raft_consensus.cc:2905] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.628634 7277 catalog_manager.cc:4143] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: term changed from 0 to 1, leader changed from to 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } } I1121 10:21:18.628826 8022 raft_consensus.cc:2359] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.628856 8240 raft_consensus.cc:469] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.628940 7774 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:18.628965 8240 raft_consensus.cc:491] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:18.629070 7774 raft_consensus.cc:2905] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.629093 8240 leader_election.cc:296] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833) I1121 10:21:18.629799 7774 raft_consensus.cc:2359] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.629833 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:18.629868 7775 raft_consensus.cc:2272] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. Re-sending same reply. I1121 10:21:18.629891 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:18.629925 7893 raft_consensus.cc:2289] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b3038ad6c564f438aa18383cddc769a in current term 1: Already voted for candidate 63b9dc8cf49b47ebb644515af2f427e7 in this term. I1121 10:21:18.629981 8363 leader_election.cc:296] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Requested vote from peers f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833) I1121 10:21:18.630304 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:18.630344 7651 raft_consensus.cc:2289] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b3038ad6c564f438aa18383cddc769a in current term 1: Already voted for candidate 63b9dc8cf49b47ebb644515af2f427e7 in this term. I1121 10:21:18.631927 7390 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" is_pre_election: true I1121 10:21:18.631961 7390 raft_consensus.cc:2320] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 1b3038ad6c564f438aa18383cddc769a for term 1 because replica is either leader or believes a valid leader to be alive. I1121 10:21:18.632019 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7d59736e513f426095bedf84bbeec48d" candidate_uuid: "1b3038ad6c564f438aa18383cddc769a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:18.632058 8022 raft_consensus.cc:2289] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b3038ad6c564f438aa18383cddc769a in current term 1: Already voted for candidate 63b9dc8cf49b47ebb644515af2f427e7 in this term. I1121 10:21:18.632122 7707 leader_election.cc:310] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a; no voters: 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d, f5bc6d2f9c574b4db9396f9f900c1141 I1121 10:21:18.632191 8240 raft_consensus.cc:2601] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I1121 10:21:18.633194 8336 raft_consensus.cc:469] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.633323 8336 raft_consensus.cc:491] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.633445 8336 leader_election.cc:296] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993) I1121 10:21:18.633615 7390 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c5abf7a6c891418482e200e7115e2c2d" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" is_pre_election: true I1121 10:21:18.633615 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c5abf7a6c891418482e200e7115e2c2d" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:18.633641 7390 raft_consensus.cc:2359] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:18.633641 8022 raft_consensus.cc:2359] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:18.633791 7831 leader_election.cc:310] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d, c9e9c35ff1cb4b68ba7cf5c037a98347; no voters: I1121 10:21:18.633790 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c5abf7a6c891418482e200e7115e2c2d" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:18.633816 7775 raft_consensus.cc:2359] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:18.633864 8336 raft_consensus.cc:2653] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:18.634047 8336 raft_consensus.cc:469] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:18.634059 8336 raft_consensus.cc:2905] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.634734 8336 raft_consensus.cc:491] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.635008 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c5abf7a6c891418482e200e7115e2c2d" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:18.635008 7390 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c5abf7a6c891418482e200e7115e2c2d" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" I1121 10:21:18.635028 8336 leader_election.cc:296] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 election: Requested vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993) I1121 10:21:18.635035 8022 raft_consensus.cc:2905] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.635035 7390 raft_consensus.cc:2905] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.635646 8022 raft_consensus.cc:2359] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:18.635646 7390 raft_consensus.cc:2359] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:18.635805 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c5abf7a6c891418482e200e7115e2c2d" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:18.635807 7831 leader_election.cc:310] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d, c9e9c35ff1cb4b68ba7cf5c037a98347; no voters: I1121 10:21:18.635843 7775 raft_consensus.cc:2905] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.635864 8336 raft_consensus.cc:2653] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:18.635926 8336 raft_consensus.cc:664] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Becoming Leader. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Running, Role: LEADER I1121 10:21:18.635996 8336 consensus_queue.cc:208] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.636390 7775 raft_consensus.cc:2359] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:18.636587 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c5abf7a6c891418482e200e7115e2c2d" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:18.636642 7651 raft_consensus.cc:2359] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:18.636724 7277 catalog_manager.cc:4143] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: term changed from 0 to 1, leader changed from to 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5). New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } } I1121 10:21:18.636596 7650 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c5abf7a6c891418482e200e7115e2c2d" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" I1121 10:21:18.636899 7650 raft_consensus.cc:2905] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.637485 7650 raft_consensus.cc:2359] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. W1121 10:21:18.702710 8385 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_1' I1121 10:21:18.731186 7390 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.731343 8022 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.731607 8022 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.731830 8363 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.731899 7650 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.731983 8328 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.732107 8299 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.732259 8328 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.732877 7402 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.733103 8273 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.733122 7775 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.733641 7893 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.733683 7527 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.733825 7775 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.733937 8291 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.734001 8273 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.734326 8273 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.734573 7651 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.734593 7893 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.734874 8363 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.735070 8286 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.735500 7774 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.735685 8299 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.738165 8364 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 37.1109 ms time per row: 0.0185555 ms I1121 10:21:18.760018 7278 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53014: name: "rebalance_test_table_2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:18.761870 7628 tablet_service.cc:1023] Processing CreateTablet for tablet 5a9a3a26473340c381d502b8e9e1660c (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.761871 7629 tablet_service.cc:1023] Processing CreateTablet for tablet 993b9cf62f6c4e6296acbb32382ab037 (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.762003 7628 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5a9a3a26473340c381d502b8e9e1660c. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.762069 7372 tablet_service.cc:1023] Processing CreateTablet for tablet 993b9cf62f6c4e6296acbb32382ab037 (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.762118 7874 tablet_service.cc:1023] Processing CreateTablet for tablet c2bd898e6d3340f5982c1386e0ebd41a (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.762173 7372 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 993b9cf62f6c4e6296acbb32382ab037. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.762205 7503 tablet_service.cc:1023] Processing CreateTablet for tablet 993b9cf62f6c4e6296acbb32382ab037 (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.762213 7874 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c2bd898e6d3340f5982c1386e0ebd41a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.762306 7503 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 993b9cf62f6c4e6296acbb32382ab037. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.762545 7370 tablet_service.cc:1023] Processing CreateTablet for tablet 5a9a3a26473340c381d502b8e9e1660c (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.762607 7370 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5a9a3a26473340c381d502b8e9e1660c. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.763320 8258 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:18.763468 7629 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 993b9cf62f6c4e6296acbb32382ab037. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.763485 7501 tablet_service.cc:1023] Processing CreateTablet for tablet 5a9a3a26473340c381d502b8e9e1660c (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.763554 7501 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5a9a3a26473340c381d502b8e9e1660c. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.763643 8258 tablet_bootstrap.cc:590] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:18.763716 7371 tablet_service.cc:1023] Processing CreateTablet for tablet c2bd898e6d3340f5982c1386e0ebd41a (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.763785 7371 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c2bd898e6d3340f5982c1386e0ebd41a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.764576 8272 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:18.764614 7507 tablet_service.cc:1023] Processing CreateTablet for tablet c2bd898e6d3340f5982c1386e0ebd41a (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.764636 8260 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:18.764703 7507 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c2bd898e6d3340f5982c1386e0ebd41a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.764922 8272 tablet_bootstrap.cc:590] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:18.764928 8260 tablet_bootstrap.cc:590] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:18.765771 7872 tablet_service.cc:1023] Processing CreateTablet for tablet 5a9a3a26473340c381d502b8e9e1660c (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.765859 7872 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5a9a3a26473340c381d502b8e9e1660c. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.766947 8002 tablet_service.cc:1023] Processing CreateTablet for tablet 993b9cf62f6c4e6296acbb32382ab037 (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.767035 8002 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 993b9cf62f6c4e6296acbb32382ab037. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.767997 8275 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:18.768023 8001 tablet_service.cc:1023] Processing CreateTablet for tablet 5a9a3a26473340c381d502b8e9e1660c (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.768088 8001 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5a9a3a26473340c381d502b8e9e1660c. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.768414 8275 tablet_bootstrap.cc:590] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:18.768596 8003 tablet_service.cc:1023] Processing CreateTablet for tablet c2bd898e6d3340f5982c1386e0ebd41a (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.768684 8003 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c2bd898e6d3340f5982c1386e0ebd41a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.769469 8259 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:18.769765 8259 tablet_bootstrap.cc:590] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:18.770103 7755 tablet_service.cc:1023] Processing CreateTablet for tablet 993b9cf62f6c4e6296acbb32382ab037 (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.770143 7754 tablet_service.cc:1023] Processing CreateTablet for tablet c2bd898e6d3340f5982c1386e0ebd41a (table=rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.770207 7755 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 993b9cf62f6c4e6296acbb32382ab037. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.770375 7754 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c2bd898e6d3340f5982c1386e0ebd41a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.771327 8264 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:18.771395 8258 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:18.771440 8258 ts_tablet_manager.cc:1104] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s I1121 10:21:18.771628 8258 raft_consensus.cc:337] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.771646 8258 raft_consensus.cc:363] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.771653 8258 raft_consensus.cc:702] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:18.771662 8264 tablet_bootstrap.cc:590] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:18.771766 8258 consensus_queue.cc:231] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.771849 8258 ts_tablet_manager.cc:1132] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.771888 8258 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:18.772209 8258 tablet_bootstrap.cc:590] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:18.772325 8272 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:18.772366 8272 ts_tablet_manager.cc:1104] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s I1121 10:21:18.772591 8272 raft_consensus.cc:337] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.772606 8272 raft_consensus.cc:363] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.772614 8272 raft_consensus.cc:702] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:18.772678 8272 consensus_queue.cc:231] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.772749 8272 ts_tablet_manager.cc:1132] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.772781 8272 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:18.772789 8259 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:18.772830 8259 ts_tablet_manager.cc:1104] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s I1121 10:21:18.772998 8259 raft_consensus.cc:337] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.773020 8259 raft_consensus.cc:363] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.773030 8259 raft_consensus.cc:702] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:18.773106 8259 consensus_queue.cc:231] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.773149 8272 tablet_bootstrap.cc:590] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:18.773190 8259 ts_tablet_manager.cc:1132] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.773241 8259 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:18.773532 8259 tablet_bootstrap.cc:590] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:18.773546 8264 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:18.773576 8264 ts_tablet_manager.cc:1104] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:18.773751 8264 raft_consensus.cc:337] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.773780 8264 raft_consensus.cc:363] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.773788 8264 raft_consensus.cc:702] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:18.773844 8272 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:18.773857 8264 consensus_queue.cc:231] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.773880 8272 ts_tablet_manager.cc:1104] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:18.773922 8264 ts_tablet_manager.cc:1132] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.773967 8264 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:18.774057 8272 raft_consensus.cc:337] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.774083 8272 raft_consensus.cc:363] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.774091 8272 raft_consensus.cc:702] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:18.774161 8272 consensus_queue.cc:231] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.774250 8272 ts_tablet_manager.cc:1132] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:18.774291 8272 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:18.774368 8264 tablet_bootstrap.cc:590] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:18.774473 8275 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:18.774521 8275 ts_tablet_manager.cc:1104] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s I1121 10:21:18.774616 8272 tablet_bootstrap.cc:590] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:18.774698 8275 raft_consensus.cc:337] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.774724 8275 raft_consensus.cc:363] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.774732 8275 raft_consensus.cc:702] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:18.774819 8275 consensus_queue.cc:231] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.774902 8275 ts_tablet_manager.cc:1132] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.774948 8275 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:18.775198 8275 tablet_bootstrap.cc:590] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:18.776022 8259 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:18.776111 8259 ts_tablet_manager.cc:1104] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s I1121 10:21:18.776248 8259 raft_consensus.cc:337] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.776273 8259 raft_consensus.cc:363] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.776281 8259 raft_consensus.cc:702] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:18.776355 8259 consensus_queue.cc:231] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.776446 8259 ts_tablet_manager.cc:1132] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.776487 8259 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:18.776832 8275 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:18.776885 8275 ts_tablet_manager.cc:1104] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1121 10:21:18.777015 8275 raft_consensus.cc:337] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.777036 8275 raft_consensus.cc:363] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.777043 8275 raft_consensus.cc:702] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:18.777104 8275 consensus_queue.cc:231] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.777189 8275 ts_tablet_manager.cc:1132] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.777226 8275 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:18.777519 8275 tablet_bootstrap.cc:590] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:18.778215 8259 tablet_bootstrap.cc:590] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:18.778734 8259 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:18.778775 8259 ts_tablet_manager.cc:1104] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:18.778920 8259 raft_consensus.cc:337] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.778944 8259 raft_consensus.cc:363] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.778950 8259 raft_consensus.cc:702] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:18.779012 8259 consensus_queue.cc:231] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.779026 8275 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:18.779038 8260 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:18.779053 8275 ts_tablet_manager.cc:1104] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1121 10:21:18.779075 8260 ts_tablet_manager.cc:1104] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s I1121 10:21:18.779214 8275 raft_consensus.cc:337] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.779235 8275 raft_consensus.cc:363] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.779243 8275 raft_consensus.cc:702] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:18.779245 8259 ts_tablet_manager.cc:1132] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.779259 8260 raft_consensus.cc:337] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.779279 8260 raft_consensus.cc:363] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.779287 8260 raft_consensus.cc:702] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:18.779322 8275 consensus_queue.cc:231] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.779359 8260 consensus_queue.cc:231] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.779438 8260 ts_tablet_manager.cc:1132] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.779490 8260 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:18.779685 8275 ts_tablet_manager.cc:1132] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:18.779788 8260 tablet_bootstrap.cc:590] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:18.780453 8272 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:18.780455 8258 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:18.780493 8272 ts_tablet_manager.cc:1104] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s I1121 10:21:18.780494 8258 ts_tablet_manager.cc:1104] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.000s I1121 10:21:18.780715 8258 raft_consensus.cc:337] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.780758 8258 raft_consensus.cc:363] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.780769 8258 raft_consensus.cc:702] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:18.780839 8258 consensus_queue.cc:231] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.780845 8272 raft_consensus.cc:337] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.780861 8272 raft_consensus.cc:363] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.780869 8272 raft_consensus.cc:702] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:18.780920 8258 ts_tablet_manager.cc:1132] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:18.780936 8272 consensus_queue.cc:231] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.781057 8272 ts_tablet_manager.cc:1132] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:18.781234 8264 tablet_bootstrap.cc:439] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:18.781285 8264 ts_tablet_manager.cc:1104] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s I1121 10:21:18.781460 8264 raft_consensus.cc:337] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.781484 8264 raft_consensus.cc:363] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.781492 8264 raft_consensus.cc:702] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:18.781560 8264 consensus_queue.cc:231] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.781657 8264 ts_tablet_manager.cc:1132] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.781859 8260 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:18.781894 8260 ts_tablet_manager.cc:1104] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:18.782029 8260 raft_consensus.cc:337] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.782050 8260 raft_consensus.cc:363] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:18.782058 8260 raft_consensus.cc:702] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:18.782227 8260 consensus_queue.cc:231] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.782303 8260 ts_tablet_manager.cc:1132] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:18.789826 8364 raft_consensus.cc:469] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.789921 8364 raft_consensus.cc:491] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.790048 8364 leader_election.cc:296] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:18.790252 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5a9a3a26473340c381d502b8e9e1660c" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:18.790256 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5a9a3a26473340c381d502b8e9e1660c" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:18.790292 7651 raft_consensus.cc:2359] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.790292 8022 raft_consensus.cc:2359] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.790414 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5a9a3a26473340c381d502b8e9e1660c" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:18.790467 7893 raft_consensus.cc:2359] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.790465 7527 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5a9a3a26473340c381d502b8e9e1660c" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:18.790493 7527 raft_consensus.cc:2359] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:18.790640 7334 leader_election.cc:310] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:18.790700 8364 raft_consensus.cc:2653] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:18.790715 8364 raft_consensus.cc:469] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:18.790724 8364 raft_consensus.cc:2905] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.791508 8364 raft_consensus.cc:491] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.791684 8364 leader_election.cc:296] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Requested vote from peers f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:18.791733 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5a9a3a26473340c381d502b8e9e1660c" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:18.791756 8022 raft_consensus.cc:2905] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.792001 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5a9a3a26473340c381d502b8e9e1660c" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" I1121 10:21:18.792037 7893 raft_consensus.cc:2905] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.792418 8022 raft_consensus.cc:2359] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.792613 7893 raft_consensus.cc:2359] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.792639 7527 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5a9a3a26473340c381d502b8e9e1660c" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:18.792660 7527 raft_consensus.cc:2905] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.792704 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5a9a3a26473340c381d502b8e9e1660c" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" I1121 10:21:18.792731 7651 raft_consensus.cc:2905] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.792747 7334 leader_election.cc:310] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d, c9e9c35ff1cb4b68ba7cf5c037a98347; no voters: I1121 10:21:18.792822 8364 raft_consensus.cc:2653] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:18.792862 8364 raft_consensus.cc:664] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Becoming Leader. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Running, Role: LEADER I1121 10:21:18.792930 8364 consensus_queue.cc:208] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.793467 7651 raft_consensus.cc:2359] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.793522 7527 raft_consensus.cc:2359] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:18.794024 7271 catalog_manager.cc:4143] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: term changed from 0 to 1, leader changed from to 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } } I1121 10:21:18.855305 8265 raft_consensus.cc:469] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.855424 8265 raft_consensus.cc:491] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.855677 8265 leader_election.cc:296] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:18.859069 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c2bd898e6d3340f5982c1386e0ebd41a" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:18.859122 7775 raft_consensus.cc:2359] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 0. I1121 10:21:18.859670 7402 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c2bd898e6d3340f5982c1386e0ebd41a" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" is_pre_election: true I1121 10:21:18.859712 7402 raft_consensus.cc:2359] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 0. I1121 10:21:18.859885 7459 leader_election.cc:310] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 41118f5c2e8f48179ffbebbfb6da2edf, 63b9dc8cf49b47ebb644515af2f427e7; no voters: I1121 10:21:18.859979 8265 raft_consensus.cc:2653] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:18.860008 8265 raft_consensus.cc:469] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:18.860015 8265 raft_consensus.cc:2905] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.860759 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c2bd898e6d3340f5982c1386e0ebd41a" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:18.860805 7893 raft_consensus.cc:2359] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 0. I1121 10:21:18.860846 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c2bd898e6d3340f5982c1386e0ebd41a" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:18.860880 8022 raft_consensus.cc:2359] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 0. I1121 10:21:18.860939 8265 raft_consensus.cc:491] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.861006 8265 leader_election.cc:296] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [CANDIDATE]: Term 1 election: Requested vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:18.861151 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c2bd898e6d3340f5982c1386e0ebd41a" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:18.861166 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c2bd898e6d3340f5982c1386e0ebd41a" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" I1121 10:21:18.861188 7893 raft_consensus.cc:2905] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.861192 8022 raft_consensus.cc:2905] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.861202 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c2bd898e6d3340f5982c1386e0ebd41a" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:18.861222 7775 raft_consensus.cc:2905] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.861776 7893 raft_consensus.cc:2359] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 1. I1121 10:21:18.861827 8022 raft_consensus.cc:2359] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 1. I1121 10:21:18.861827 7775 raft_consensus.cc:2359] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 1. I1121 10:21:18.861923 7402 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c2bd898e6d3340f5982c1386e0ebd41a" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" I1121 10:21:18.861954 7402 raft_consensus.cc:2905] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.861968 7460 leader_election.cc:310] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 41118f5c2e8f48179ffbebbfb6da2edf, 81d8bacc3ca64d628a255ad416a77c3d; no voters: I1121 10:21:18.862040 8265 raft_consensus.cc:2653] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:18.862076 8265 raft_consensus.cc:664] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 LEADER]: Becoming Leader. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Running, Role: LEADER I1121 10:21:18.862164 8265 consensus_queue.cc:208] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:18.862570 7402 raft_consensus.cc:2359] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 1. I1121 10:21:18.862951 7271 catalog_manager.cc:4143] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf reported cstate change: term changed from 0 to 1, leader changed from to 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2). New cstate: current_term: 1 leader_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } } I1121 10:21:18.870103 8400 raft_consensus.cc:469] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:18.870254 8400 raft_consensus.cc:491] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.870360 8400 leader_election.cc:296] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995) I1121 10:21:18.870563 7527 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "993b9cf62f6c4e6296acbb32382ab037" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:18.870560 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "993b9cf62f6c4e6296acbb32382ab037" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:18.870589 7527 raft_consensus.cc:2359] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.870589 7775 raft_consensus.cc:2359] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.870688 7402 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "993b9cf62f6c4e6296acbb32382ab037" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" is_pre_election: true I1121 10:21:18.870704 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "993b9cf62f6c4e6296acbb32382ab037" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:18.870718 7402 raft_consensus.cc:2359] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.870731 8022 raft_consensus.cc:2359] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:18.870746 7584 leader_election.cc:310] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 41118f5c2e8f48179ffbebbfb6da2edf, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:18.870805 8400 raft_consensus.cc:2653] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:18.870816 8400 raft_consensus.cc:469] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:18.870824 8400 raft_consensus.cc:2905] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.871462 8400 raft_consensus.cc:491] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.871533 8400 leader_election.cc:296] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 election: Requested vote from peers 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995) I1121 10:21:18.871696 7402 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "993b9cf62f6c4e6296acbb32382ab037" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" I1121 10:21:18.871727 7402 raft_consensus.cc:2905] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.871739 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "993b9cf62f6c4e6296acbb32382ab037" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:18.871768 8022 raft_consensus.cc:2905] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.871776 7775 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "993b9cf62f6c4e6296acbb32382ab037" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:18.871806 7775 raft_consensus.cc:2905] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.872236 8022 raft_consensus.cc:2359] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.872400 7775 raft_consensus.cc:2359] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.872400 7402 raft_consensus.cc:2359] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.872493 7527 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "993b9cf62f6c4e6296acbb32382ab037" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:18.872552 7527 raft_consensus.cc:2905] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:18.872556 7583 leader_election.cc:310] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, c9e9c35ff1cb4b68ba7cf5c037a98347, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:18.872611 8400 raft_consensus.cc:2653] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:18.872682 8400 raft_consensus.cc:664] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Becoming Leader. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Running, Role: LEADER I1121 10:21:18.872763 8400 consensus_queue.cc:208] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } I1121 10:21:18.873476 7527 raft_consensus.cc:2359] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:18.873616 7271 catalog_manager.cc:4143] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: term changed from 0 to 1, leader changed from to f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } } W1121 10:21:18.939270 8438 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_2' I1121 10:21:18.970127 7402 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.970213 8022 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.970278 7527 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.970314 8023 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.970438 8273 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.970484 7893 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.970559 7896 raft_consensus.cc:1181] T c2bd898e6d3340f5982c1386e0ebd41a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.970618 7775 raft_consensus.cc:1181] T c2bd898e6d3340f5982c1386e0ebd41a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.970683 8023 raft_consensus.cc:1181] T c2bd898e6d3340f5982c1386e0ebd41a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.970788 8265 consensus_queue.cc:984] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.970829 8286 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.970839 8273 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.971014 7402 raft_consensus.cc:1181] T c2bd898e6d3340f5982c1386e0ebd41a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.971048 7651 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.971189 8307 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.970623 7773 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.971271 8268 consensus_queue.cc:984] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.971387 7527 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:18.971405 8268 consensus_queue.cc:984] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.971469 8286 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.971508 8268 consensus_queue.cc:984] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.971664 8286 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.973716 8291 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:18.973879 8291 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 34.1568 ms time per row: 0.0170784 ms I1121 10:21:18.995317 7271 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53014: name: "rebalance_test_table_3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:18.996826 7629 tablet_service.cc:1023] Processing CreateTablet for tablet 7cd3c7862a7c420cb0b87bd87a770d84 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.996903 8001 tablet_service.cc:1023] Processing CreateTablet for tablet 7cd3c7862a7c420cb0b87bd87a770d84 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.996937 7872 tablet_service.cc:1023] Processing CreateTablet for tablet 7cd3c7862a7c420cb0b87bd87a770d84 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.996950 7629 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7cd3c7862a7c420cb0b87bd87a770d84. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.997004 8001 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7cd3c7862a7c420cb0b87bd87a770d84. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.997026 7872 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7cd3c7862a7c420cb0b87bd87a770d84. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.997403 7628 tablet_service.cc:1023] Processing CreateTablet for tablet 169c4cf587a949748d1734c65258b7a7 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.997485 7628 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 169c4cf587a949748d1734c65258b7a7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.998234 7753 tablet_service.cc:1023] Processing CreateTablet for tablet 865e88694908474197879cd8d34789ad (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.998298 8258 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:18.998328 7753 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 865e88694908474197879cd8d34789ad. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.998356 7755 tablet_service.cc:1023] Processing CreateTablet for tablet 169c4cf587a949748d1734c65258b7a7 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.998477 7874 tablet_service.cc:1023] Processing CreateTablet for tablet 169c4cf587a949748d1734c65258b7a7 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.998558 7874 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 169c4cf587a949748d1734c65258b7a7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.998600 8258 tablet_bootstrap.cc:590] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:18.998653 7627 tablet_service.cc:1023] Processing CreateTablet for tablet 865e88694908474197879cd8d34789ad (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.998741 7627 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 865e88694908474197879cd8d34789ad. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.999090 8260 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:18.999425 8260 tablet_bootstrap.cc:590] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:18.998234 7754 tablet_service.cc:1023] Processing CreateTablet for tablet 7cd3c7862a7c420cb0b87bd87a770d84 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.999693 7755 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 169c4cf587a949748d1734c65258b7a7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:18.999724 7871 tablet_service.cc:1023] Processing CreateTablet for tablet 865e88694908474197879cd8d34789ad (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:18.999794 7871 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 865e88694908474197879cd8d34789ad. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.000193 7501 tablet_service.cc:1023] Processing CreateTablet for tablet 7cd3c7862a7c420cb0b87bd87a770d84 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.000262 8264 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:19.000288 7501 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7cd3c7862a7c420cb0b87bd87a770d84. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.000499 8264 tablet_bootstrap.cc:590] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:19.000600 7507 tablet_service.cc:1023] Processing CreateTablet for tablet 169c4cf587a949748d1734c65258b7a7 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.000599 8275 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:19.000667 7507 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 169c4cf587a949748d1734c65258b7a7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.000926 8275 tablet_bootstrap.cc:590] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:19.001401 7371 tablet_service.cc:1023] Processing CreateTablet for tablet 865e88694908474197879cd8d34789ad (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.001473 7371 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 865e88694908474197879cd8d34789ad. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.001796 7503 tablet_service.cc:1023] Processing CreateTablet for tablet 865e88694908474197879cd8d34789ad (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.000586 7754 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7cd3c7862a7c420cb0b87bd87a770d84. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.001868 7503 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 865e88694908474197879cd8d34789ad. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.002317 8272 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:19.002655 8272 tablet_bootstrap.cc:590] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:19.002882 8259 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:19.002972 8001 tablet_service.cc:1023] Processing CreateTablet for tablet 169c4cf587a949748d1734c65258b7a7 (table=rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.003060 8001 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 169c4cf587a949748d1734c65258b7a7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.003209 8259 tablet_bootstrap.cc:590] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:19.004927 8260 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:19.004979 8260 ts_tablet_manager.cc:1104] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s I1121 10:21:19.005053 8275 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:19.005096 8275 ts_tablet_manager.cc:1104] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s I1121 10:21:19.005167 8260 raft_consensus.cc:337] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.005188 8260 raft_consensus.cc:363] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.005194 8260 raft_consensus.cc:702] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:19.005268 8260 consensus_queue.cc:231] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.005270 8275 raft_consensus.cc:337] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.005296 8275 raft_consensus.cc:363] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.005306 8275 raft_consensus.cc:702] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:19.005349 8260 ts_tablet_manager.cc:1132] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.005373 8275 consensus_queue.cc:231] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.005388 8260 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:19.005445 8275 ts_tablet_manager.cc:1132] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.005479 8275 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:19.005751 8260 tablet_bootstrap.cc:590] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:19.005810 8275 tablet_bootstrap.cc:590] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:19.006162 8272 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:19.006228 8272 ts_tablet_manager.cc:1104] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s I1121 10:21:19.006263 8258 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:19.006338 8258 ts_tablet_manager.cc:1104] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s I1121 10:21:19.006423 8272 raft_consensus.cc:337] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.006449 8272 raft_consensus.cc:363] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.006458 8272 raft_consensus.cc:702] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:19.006517 8258 raft_consensus.cc:337] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.006527 8272 consensus_queue.cc:231] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.006536 8258 raft_consensus.cc:363] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.006546 8258 raft_consensus.cc:702] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:19.006599 8272 ts_tablet_manager.cc:1132] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.006613 8258 consensus_queue.cc:231] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.006647 8259 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:19.006675 8259 ts_tablet_manager.cc:1104] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s I1121 10:21:19.006682 8258 ts_tablet_manager.cc:1132] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:19.006724 8258 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:19.006848 8259 raft_consensus.cc:337] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.006865 8259 raft_consensus.cc:363] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.006883 8259 raft_consensus.cc:702] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:19.006952 8259 consensus_queue.cc:231] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.007023 8259 ts_tablet_manager.cc:1132] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.007063 8258 tablet_bootstrap.cc:590] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:19.007076 8259 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:19.007382 8259 tablet_bootstrap.cc:590] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:19.007443 8275 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:19.007473 8275 ts_tablet_manager.cc:1104] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:19.007611 8258 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:19.007642 8258 ts_tablet_manager.cc:1104] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:19.007684 8275 raft_consensus.cc:337] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.007700 8275 raft_consensus.cc:363] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.007707 8275 raft_consensus.cc:702] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:19.007799 8275 consensus_queue.cc:231] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.007802 8258 raft_consensus.cc:337] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.007817 8258 raft_consensus.cc:363] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.007825 8258 raft_consensus.cc:702] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:19.007869 8275 ts_tablet_manager.cc:1132] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.007910 8258 consensus_queue.cc:231] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.007975 8258 ts_tablet_manager.cc:1132] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.008008 8258 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:19.008330 8260 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:19.008332 8258 tablet_bootstrap.cc:590] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:19.008368 8260 ts_tablet_manager.cc:1104] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:19.008539 8260 raft_consensus.cc:337] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.008563 8260 raft_consensus.cc:363] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.008570 8260 raft_consensus.cc:702] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:19.008633 8260 consensus_queue.cc:231] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.008698 8260 ts_tablet_manager.cc:1132] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.008762 8260 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:19.009196 8260 tablet_bootstrap.cc:590] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:19.009224 8259 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:19.009253 8259 ts_tablet_manager.cc:1104] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:19.009414 8259 raft_consensus.cc:337] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.009428 8259 raft_consensus.cc:363] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.009434 8259 raft_consensus.cc:702] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:19.009500 8259 consensus_queue.cc:231] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.009575 8259 ts_tablet_manager.cc:1132] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.009611 8259 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:19.009783 8258 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:19.009814 8258 ts_tablet_manager.cc:1104] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:19.009920 8259 tablet_bootstrap.cc:590] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:19.009943 8258 raft_consensus.cc:337] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.009961 8258 raft_consensus.cc:363] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.009968 8258 raft_consensus.cc:702] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:19.010012 8264 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:19.010033 8258 consensus_queue.cc:231] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.010054 8264 ts_tablet_manager.cc:1104] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s I1121 10:21:19.010097 8258 ts_tablet_manager.cc:1132] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.010215 8264 raft_consensus.cc:337] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.010229 8264 raft_consensus.cc:363] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.010239 8264 raft_consensus.cc:702] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:19.010323 8264 consensus_queue.cc:231] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.010408 8264 ts_tablet_manager.cc:1132] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.010449 8264 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:19.010789 8264 tablet_bootstrap.cc:590] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:19.011484 8264 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:19.011520 8264 ts_tablet_manager.cc:1104] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:19.011649 8259 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:19.011682 8259 ts_tablet_manager.cc:1104] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1121 10:21:19.011699 8264 raft_consensus.cc:337] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.011713 8264 raft_consensus.cc:363] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.011720 8264 raft_consensus.cc:702] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:19.011785 8264 consensus_queue.cc:231] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.011838 8259 raft_consensus.cc:337] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.011852 8264 ts_tablet_manager.cc:1132] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.011857 8259 raft_consensus.cc:363] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.011864 8259 raft_consensus.cc:702] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:19.011886 8264 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:19.011924 8259 consensus_queue.cc:231] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.012161 8259 ts_tablet_manager.cc:1132] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.012176 8264 tablet_bootstrap.cc:590] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:19.024893 8336 raft_consensus.cc:469] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:19.024982 8336 raft_consensus.cc:491] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.025095 8336 leader_election.cc:296] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833) I1121 10:21:19.025310 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:19.025312 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:19.025346 7651 raft_consensus.cc:2359] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:19.025349 8023 raft_consensus.cc:2359] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:19.025357 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:19.025389 7519 raft_consensus.cc:2359] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:19.025384 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:19.025499 7831 leader_election.cc:310] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 81d8bacc3ca64d628a255ad416a77c3d, c9e9c35ff1cb4b68ba7cf5c037a98347, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:19.025604 8336 raft_consensus.cc:2653] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:19.025622 8336 raft_consensus.cc:469] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:19.025630 8336 raft_consensus.cc:2905] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 W1121 10:21:19.025776 7832 leader_election.cc:348] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993): Illegal state: must be running to vote when last-logged opid is not known I1121 10:21:19.028647 8260 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:19.028697 8260 ts_tablet_manager.cc:1104] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.000s I1121 10:21:19.028913 8260 raft_consensus.cc:337] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.028934 8260 raft_consensus.cc:363] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.028941 8260 raft_consensus.cc:702] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:19.029011 8260 consensus_queue.cc:231] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.029099 8260 ts_tablet_manager.cc:1132] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.029278 8336 raft_consensus.cc:491] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.029387 8336 leader_election.cc:296] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 election: Requested vote from peers f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833) I1121 10:21:19.029552 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" I1121 10:21:19.029569 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:19.029577 7651 raft_consensus.cc:2905] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.029597 8023 raft_consensus.cc:2905] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.029628 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:19.029687 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:19.029775 7519 raft_consensus.cc:2905] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 W1121 10:21:19.029783 7832 leader_election.cc:348] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993): Illegal state: must be running to vote when last-logged opid is not known I1121 10:21:19.030192 8023 raft_consensus.cc:2359] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:19.040664 8264 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:19.040719 8264 ts_tablet_manager.cc:1104] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.000s I1121 10:21:19.040968 8264 raft_consensus.cc:337] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.040992 8264 raft_consensus.cc:363] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.040997 8264 raft_consensus.cc:702] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:19.041066 8264 consensus_queue.cc:231] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.041180 8264 ts_tablet_manager.cc:1132] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.049187 7519 raft_consensus.cc:2359] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:19.049327 7651 raft_consensus.cc:2359] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:19.049401 7832 leader_election.cc:310] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, 81d8bacc3ca64d628a255ad416a77c3d, c9e9c35ff1cb4b68ba7cf5c037a98347; no voters: 1b3038ad6c564f438aa18383cddc769a I1121 10:21:19.049566 8336 raft_consensus.cc:2653] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:19.049614 8336 raft_consensus.cc:664] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Becoming Leader. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Running, Role: LEADER I1121 10:21:19.049706 8336 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.050431 7275 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: term changed from 0 to 1, leader changed from to 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5). New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } } I1121 10:21:19.127445 8273 raft_consensus.cc:469] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:19.127570 8273 raft_consensus.cc:491] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.127718 8273 leader_election.cc:296] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833) I1121 10:21:19.127948 7896 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "169c4cf587a949748d1734c65258b7a7" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:19.127970 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "169c4cf587a949748d1734c65258b7a7" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:19.127974 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "169c4cf587a949748d1734c65258b7a7" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:19.127985 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "169c4cf587a949748d1734c65258b7a7" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:19.128001 7896 raft_consensus.cc:2359] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:19.128006 8023 raft_consensus.cc:2359] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:19.128010 7773 raft_consensus.cc:2359] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:19.128018 7519 raft_consensus.cc:2359] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 0. I1121 10:21:19.128170 7583 leader_election.cc:310] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, c9e9c35ff1cb4b68ba7cf5c037a98347, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:19.128304 8273 raft_consensus.cc:2653] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:19.128325 8273 raft_consensus.cc:469] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:19.128331 8273 raft_consensus.cc:2905] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.129109 8273 raft_consensus.cc:491] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.129194 8273 leader_election.cc:296] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 election: Requested vote from peers 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833) I1121 10:21:19.129406 8023 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "169c4cf587a949748d1734c65258b7a7" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:19.129417 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "169c4cf587a949748d1734c65258b7a7" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:19.129431 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "169c4cf587a949748d1734c65258b7a7" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:19.129436 8023 raft_consensus.cc:2905] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.129443 7519 raft_consensus.cc:2905] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.129448 7773 raft_consensus.cc:2905] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.129447 7896 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "169c4cf587a949748d1734c65258b7a7" candidate_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" I1121 10:21:19.129468 7896 raft_consensus.cc:2905] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.130156 7773 raft_consensus.cc:2359] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:19.130162 7896 raft_consensus.cc:2359] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:19.130164 8023 raft_consensus.cc:2359] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:19.130168 7519 raft_consensus.cc:2359] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5bc6d2f9c574b4db9396f9f900c1141 in term 1. I1121 10:21:19.130340 7584 leader_election.cc:310] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, c9e9c35ff1cb4b68ba7cf5c037a98347, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:19.130427 8273 raft_consensus.cc:2653] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:19.130466 8273 raft_consensus.cc:664] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Becoming Leader. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Running, Role: LEADER I1121 10:21:19.130548 8273 consensus_queue.cc:208] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } I1121 10:21:19.131211 7275 catalog_manager.cc:4143] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: term changed from 0 to 1, leader changed from to f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } } I1121 10:21:19.135123 8454 raft_consensus.cc:469] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:19.135215 8454 raft_consensus.cc:491] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.135378 8454 leader_election.cc:296] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:19.135601 7402 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "865e88694908474197879cd8d34789ad" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" is_pre_election: true I1121 10:21:19.135612 7896 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "865e88694908474197879cd8d34789ad" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:19.135640 7896 raft_consensus.cc:2359] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 0. I1121 10:21:19.135641 7402 raft_consensus.cc:2359] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 0. I1121 10:21:19.135674 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "865e88694908474197879cd8d34789ad" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:19.135704 7773 raft_consensus.cc:2359] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 0. I1121 10:21:19.135854 7459 leader_election.cc:310] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d; no voters: I1121 10:21:19.135941 8454 raft_consensus.cc:2653] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:19.135962 8454 raft_consensus.cc:469] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:19.135989 8454 raft_consensus.cc:2905] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.136647 8454 raft_consensus.cc:491] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.136751 8454 leader_election.cc:296] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [CANDIDATE]: Term 1 election: Requested vote from peers 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909), 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:19.136970 7402 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "865e88694908474197879cd8d34789ad" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" I1121 10:21:19.136971 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "865e88694908474197879cd8d34789ad" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:19.137006 7773 raft_consensus.cc:2905] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.137006 7402 raft_consensus.cc:2905] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.137615 7773 raft_consensus.cc:2359] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 1. I1121 10:21:19.137763 7402 raft_consensus.cc:2359] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 1. I1121 10:21:19.137910 7896 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "865e88694908474197879cd8d34789ad" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" I1121 10:21:19.137933 7896 raft_consensus.cc:2905] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.138054 7651 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "865e88694908474197879cd8d34789ad" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:19.138092 7651 raft_consensus.cc:2359] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 0. I1121 10:21:19.138113 7650 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "865e88694908474197879cd8d34789ad" candidate_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" I1121 10:21:19.138145 7650 raft_consensus.cc:2905] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.138520 7896 raft_consensus.cc:2359] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 1. I1121 10:21:19.138604 7459 leader_election.cc:310] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 41118f5c2e8f48179ffbebbfb6da2edf, 63b9dc8cf49b47ebb644515af2f427e7; no voters: I1121 10:21:19.138682 8454 raft_consensus.cc:2653] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:19.138726 8454 raft_consensus.cc:664] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 LEADER]: Becoming Leader. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Running, Role: LEADER I1121 10:21:19.138741 7650 raft_consensus.cc:2359] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41118f5c2e8f48179ffbebbfb6da2edf in term 1. I1121 10:21:19.138821 8454 consensus_queue.cc:208] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.139457 7275 catalog_manager.cc:4143] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf reported cstate change: term changed from 0 to 1, leader changed from to 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2). New cstate: current_term: 1 leader_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } } W1121 10:21:19.202687 8486 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_3' I1121 10:21:19.231302 8023 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:19.231544 8273 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.232023 7773 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:19.232151 7650 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.232219 7896 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:19.232338 7519 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:19.232383 7402 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.232429 7527 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.232496 8265 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.232595 8022 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.232827 8328 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.232916 8328 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.233259 8501 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.233294 8291 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.233323 7651 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.233427 8347 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.233485 7773 raft_consensus.cc:2905] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.233592 8291 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.234331 7773 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.235757 8266 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.235785 7893 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.235983 8501 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.236069 7773 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.237954 8328 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.238602 8265 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 34.9851 ms time per row: 0.0174926 ms I1121 10:21:19.257119 7274 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53014: name: "rebalance_test_table_4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:19.258664 7501 tablet_service.cc:1023] Processing CreateTablet for tablet 55d18b1e7ede4cb28f5dfc4ded166a30 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.258672 7754 tablet_service.cc:1023] Processing CreateTablet for tablet 55d18b1e7ede4cb28f5dfc4ded166a30 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.258709 7371 tablet_service.cc:1023] Processing CreateTablet for tablet 55d18b1e7ede4cb28f5dfc4ded166a30 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.258787 8001 tablet_service.cc:1023] Processing CreateTablet for tablet 55d18b1e7ede4cb28f5dfc4ded166a30 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.258798 7754 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 55d18b1e7ede4cb28f5dfc4ded166a30. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.258818 7371 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 55d18b1e7ede4cb28f5dfc4ded166a30. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.258842 7501 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 55d18b1e7ede4cb28f5dfc4ded166a30. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.258877 8001 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 55d18b1e7ede4cb28f5dfc4ded166a30. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.260051 8264 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:19.260069 8003 tablet_service.cc:1023] Processing CreateTablet for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.260074 8259 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:19.260095 7372 tablet_service.cc:1023] Processing CreateTablet for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.260140 8003 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5c8cdc07bf6f4480b269ca34a9b3e426. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.260165 7372 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5c8cdc07bf6f4480b269ca34a9b3e426. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.260483 8264 tablet_bootstrap.cc:590] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:19.260599 7370 tablet_service.cc:1023] Processing CreateTablet for tablet f30bc779dfa840c6ae504b94d82eaa5a (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.260607 8002 tablet_service.cc:1023] Processing CreateTablet for tablet f30bc779dfa840c6ae504b94d82eaa5a (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.260620 8259 tablet_bootstrap.cc:590] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:19.260674 7370 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet f30bc779dfa840c6ae504b94d82eaa5a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.260679 8002 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet f30bc779dfa840c6ae504b94d82eaa5a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.260753 8272 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:19.260880 7871 tablet_service.cc:1023] Processing CreateTablet for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.260967 7871 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5c8cdc07bf6f4480b269ca34a9b3e426. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.261098 7503 tablet_service.cc:1023] Processing CreateTablet for tablet f30bc779dfa840c6ae504b94d82eaa5a (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.261180 7503 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet f30bc779dfa840c6ae504b94d82eaa5a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.261665 8275 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:19.261998 8275 tablet_bootstrap.cc:590] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:19.262164 7874 tablet_service.cc:1023] Processing CreateTablet for tablet f30bc779dfa840c6ae504b94d82eaa5a (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.262250 7874 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet f30bc779dfa840c6ae504b94d82eaa5a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.262289 8259 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:19.262321 8259 ts_tablet_manager.cc:1104] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:19.262378 7501 tablet_service.cc:1023] Processing CreateTablet for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.262444 7501 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5c8cdc07bf6f4480b269ca34a9b3e426. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.262501 8259 raft_consensus.cc:337] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.262516 8259 raft_consensus.cc:363] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.262535 8259 raft_consensus.cc:702] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:19.262600 8259 consensus_queue.cc:231] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.262673 8259 ts_tablet_manager.cc:1132] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.262715 8259 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:19.263032 8259 tablet_bootstrap.cc:590] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:19.263267 8260 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:19.263528 8260 tablet_bootstrap.cc:590] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:19.263803 8275 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:19.263839 8275 ts_tablet_manager.cc:1104] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:19.263917 8259 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:19.263947 8259 ts_tablet_manager.cc:1104] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:19.264001 8275 raft_consensus.cc:337] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.264017 8275 raft_consensus.cc:363] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.264025 8275 raft_consensus.cc:702] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:19.264061 8260 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:19.264080 8259 raft_consensus.cc:337] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.264091 8260 ts_tablet_manager.cc:1104] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:19.264093 8259 raft_consensus.cc:363] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.264101 8259 raft_consensus.cc:702] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:19.264098 8275 consensus_queue.cc:231] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.264169 8259 consensus_queue.cc:231] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.264179 8275 ts_tablet_manager.cc:1132] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.264214 8275 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:19.264235 8259 ts_tablet_manager.cc:1132] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.264256 8260 raft_consensus.cc:337] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.264271 8260 raft_consensus.cc:363] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.264277 8260 raft_consensus.cc:702] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:19.264276 8259 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:19.264354 8260 consensus_queue.cc:231] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.264431 8260 ts_tablet_manager.cc:1132] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.264463 8260 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d: Bootstrap starting. I1121 10:21:19.264657 8259 tablet_bootstrap.cc:590] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: No blocks or log segments found. Creating new log. I1121 10:21:19.264678 7627 tablet_service.cc:1023] Processing CreateTablet for tablet 55d18b1e7ede4cb28f5dfc4ded166a30 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.264789 7627 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 55d18b1e7ede4cb28f5dfc4ded166a30. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.265128 8275 tablet_bootstrap.cc:590] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:19.265308 8260 tablet_bootstrap.cc:590] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d: No blocks or log segments found. Creating new log. I1121 10:21:19.265873 7628 tablet_service.cc:1023] Processing CreateTablet for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.265965 7628 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5c8cdc07bf6f4480b269ca34a9b3e426. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.265988 8260 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d: No bootstrap required, opened a new log I1121 10:21:19.266023 8260 ts_tablet_manager.cc:1104] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:19.266134 8259 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: No bootstrap required, opened a new log I1121 10:21:19.266173 8259 ts_tablet_manager.cc:1104] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:19.266176 8260 raft_consensus.cc:337] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.266199 8260 raft_consensus.cc:363] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.266229 8260 raft_consensus.cc:702] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Initialized, Role: FOLLOWER I1121 10:21:19.266305 8260 consensus_queue.cc:231] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.266316 8259 raft_consensus.cc:337] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.266330 8259 raft_consensus.cc:363] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.266337 8259 raft_consensus.cc:702] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: FOLLOWER I1121 10:21:19.266374 8260 ts_tablet_manager.cc:1132] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.266423 8259 consensus_queue.cc:231] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.266490 8259 ts_tablet_manager.cc:1132] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.266796 8264 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:19.266855 8264 ts_tablet_manager.cc:1104] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s I1121 10:21:19.266903 8258 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:19.267030 8264 raft_consensus.cc:337] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.267060 8264 raft_consensus.cc:363] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.267069 8264 raft_consensus.cc:702] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:19.267134 8264 consensus_queue.cc:231] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.267212 8258 tablet_bootstrap.cc:590] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:19.267223 8264 ts_tablet_manager.cc:1132] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s I1121 10:21:19.267674 7754 tablet_service.cc:1023] Processing CreateTablet for tablet f30bc779dfa840c6ae504b94d82eaa5a (table=rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:19.267737 8258 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:19.267765 7754 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet f30bc779dfa840c6ae504b94d82eaa5a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:19.267768 8258 ts_tablet_manager.cc:1104] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:19.267905 8258 raft_consensus.cc:337] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.267915 8258 raft_consensus.cc:363] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.267920 8258 raft_consensus.cc:702] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:19.267973 8258 consensus_queue.cc:231] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.268044 8258 ts_tablet_manager.cc:1132] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.268074 8258 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141: Bootstrap starting. I1121 10:21:19.268430 8258 tablet_bootstrap.cc:590] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141: No blocks or log segments found. Creating new log. I1121 10:21:19.269119 8264 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a: Bootstrap starting. I1121 10:21:19.269156 8258 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141: No bootstrap required, opened a new log I1121 10:21:19.269194 8258 ts_tablet_manager.cc:1104] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:19.269335 8258 raft_consensus.cc:337] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.269356 8258 raft_consensus.cc:363] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.269371 8258 raft_consensus.cc:702] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5bc6d2f9c574b4db9396f9f900c1141, State: Initialized, Role: FOLLOWER I1121 10:21:19.269436 8258 consensus_queue.cc:231] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.269443 8264 tablet_bootstrap.cc:590] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a: No blocks or log segments found. Creating new log. I1121 10:21:19.269501 8258 ts_tablet_manager.cc:1132] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.269963 8264 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a: No bootstrap required, opened a new log I1121 10:21:19.269994 8264 ts_tablet_manager.cc:1104] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:19.270144 8264 raft_consensus.cc:337] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.270167 8264 raft_consensus.cc:363] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.270175 8264 raft_consensus.cc:702] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3038ad6c564f438aa18383cddc769a, State: Initialized, Role: FOLLOWER I1121 10:21:19.270260 8264 consensus_queue.cc:231] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.270349 8264 ts_tablet_manager.cc:1132] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.284714 8275 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:19.284726 8272 tablet_bootstrap.cc:590] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:19.284790 8275 ts_tablet_manager.cc:1104] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.000s I1121 10:21:19.285037 8275 raft_consensus.cc:337] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.285063 8275 raft_consensus.cc:363] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.285069 8275 raft_consensus.cc:702] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:19.285145 8275 consensus_queue.cc:231] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.285239 8275 ts_tablet_manager.cc:1132] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:19.285290 8275 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347: Bootstrap starting. I1121 10:21:19.285611 8272 tablet_bootstrap.cc:439] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:19.285640 8275 tablet_bootstrap.cc:590] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347: No blocks or log segments found. Creating new log. I1121 10:21:19.285661 8272 ts_tablet_manager.cc:1104] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.000s I1121 10:21:19.285832 8272 raft_consensus.cc:337] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.285854 8272 raft_consensus.cc:363] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.285861 8272 raft_consensus.cc:702] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:19.285918 8272 consensus_queue.cc:231] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.285997 8272 ts_tablet_manager.cc:1132] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.286048 8272 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:19.286471 8272 tablet_bootstrap.cc:590] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:19.287060 8275 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347: No bootstrap required, opened a new log I1121 10:21:19.287091 8272 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:19.287101 8275 ts_tablet_manager.cc:1104] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:19.287129 8272 ts_tablet_manager.cc:1104] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:19.287252 8275 raft_consensus.cc:337] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.287254 8272 raft_consensus.cc:337] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.287272 8272 raft_consensus.cc:363] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.287274 8275 raft_consensus.cc:363] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.287277 8272 raft_consensus.cc:702] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:19.287281 8275 raft_consensus.cc:702] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9c35ff1cb4b68ba7cf5c037a98347, State: Initialized, Role: FOLLOWER I1121 10:21:19.287353 8272 consensus_queue.cc:231] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.287369 8275 consensus_queue.cc:231] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.287436 8272 ts_tablet_manager.cc:1132] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.287479 8272 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:19.287612 8275 ts_tablet_manager.cc:1132] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.287899 8272 tablet_bootstrap.cc:590] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7: No blocks or log segments found. Creating new log. I1121 10:21:19.288450 8272 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7: No bootstrap required, opened a new log I1121 10:21:19.288493 8272 ts_tablet_manager.cc:1104] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:19.288671 8272 raft_consensus.cc:337] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.288693 8272 raft_consensus.cc:363] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:19.288734 8272 raft_consensus.cc:702] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: FOLLOWER I1121 10:21:19.288808 8272 consensus_queue.cc:231] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.288889 8272 ts_tablet_manager.cc:1132] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.290138 8363 raft_consensus.cc:469] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:19.290197 8363 raft_consensus.cc:491] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.290345 8363 leader_election.cc:296] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:19.290509 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "55d18b1e7ede4cb28f5dfc4ded166a30" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:19.290539 8022 raft_consensus.cc:2359] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:19.290551 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "55d18b1e7ede4cb28f5dfc4ded166a30" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:19.290550 7650 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "55d18b1e7ede4cb28f5dfc4ded166a30" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:19.290585 7650 raft_consensus.cc:2359] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:19.290585 7519 raft_consensus.cc:2359] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:19.290715 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "55d18b1e7ede4cb28f5dfc4ded166a30" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:19.290777 7773 raft_consensus.cc:2359] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:19.290915 7333 leader_election.cc:310] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, c9e9c35ff1cb4b68ba7cf5c037a98347, f5bc6d2f9c574b4db9396f9f900c1141; no voters: I1121 10:21:19.291002 8363 raft_consensus.cc:2653] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:19.291023 8363 raft_consensus.cc:469] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:19.291049 8363 raft_consensus.cc:2905] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.291800 8363 raft_consensus.cc:491] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.291883 8363 leader_election.cc:296] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Requested vote from peers 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:19.292043 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "55d18b1e7ede4cb28f5dfc4ded166a30" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:19.292044 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "55d18b1e7ede4cb28f5dfc4ded166a30" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:19.292045 7650 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "55d18b1e7ede4cb28f5dfc4ded166a30" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" I1121 10:21:19.292071 8022 raft_consensus.cc:2905] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.292071 7650 raft_consensus.cc:2905] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.292073 7773 raft_consensus.cc:2905] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.292603 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "55d18b1e7ede4cb28f5dfc4ded166a30" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:19.292650 7519 raft_consensus.cc:2905] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.292824 7773 raft_consensus.cc:2359] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:19.292883 8022 raft_consensus.cc:2359] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:19.293035 7333 leader_election.cc:310] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1b3038ad6c564f438aa18383cddc769a, 63b9dc8cf49b47ebb644515af2f427e7, c9e9c35ff1cb4b68ba7cf5c037a98347; no voters: I1121 10:21:19.293103 8363 raft_consensus.cc:2653] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:19.293135 8363 raft_consensus.cc:664] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Becoming Leader. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Running, Role: LEADER I1121 10:21:19.293210 8363 consensus_queue.cc:208] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.293359 7519 raft_consensus.cc:2359] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:19.293686 7650 raft_consensus.cc:2359] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:19.293913 7274 catalog_manager.cc:4143] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: term changed from 0 to 1, leader changed from to 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } } I1121 10:21:19.369704 8502 raft_consensus.cc:469] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:19.369832 8502 raft_consensus.cc:491] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.370105 8502 leader_election.cc:296] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:19.370112 7402 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5c8cdc07bf6f4480b269ca34a9b3e426" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" is_pre_election: true I1121 10:21:19.370155 7402 raft_consensus.cc:2359] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:19.370297 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5c8cdc07bf6f4480b269ca34a9b3e426" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:19.370357 7519 raft_consensus.cc:2359] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:19.370416 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5c8cdc07bf6f4480b269ca34a9b3e426" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:19.370453 8022 raft_consensus.cc:2359] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:19.370488 7650 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5c8cdc07bf6f4480b269ca34a9b3e426" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" is_pre_election: true I1121 10:21:19.370496 7832 leader_election.cc:310] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d; no voters: I1121 10:21:19.370527 7650 raft_consensus.cc:2359] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 0. I1121 10:21:19.370568 8502 raft_consensus.cc:2653] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:19.370581 8502 raft_consensus.cc:469] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:19.370589 8502 raft_consensus.cc:2905] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.371353 8502 raft_consensus.cc:491] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.371598 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5c8cdc07bf6f4480b269ca34a9b3e426" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:19.371608 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5c8cdc07bf6f4480b269ca34a9b3e426" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:19.371623 7402 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5c8cdc07bf6f4480b269ca34a9b3e426" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63b9dc8cf49b47ebb644515af2f427e7" I1121 10:21:19.371628 8022 raft_consensus.cc:2905] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.371631 7519 raft_consensus.cc:2905] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.371644 7402 raft_consensus.cc:2905] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.372120 7519 raft_consensus.cc:2359] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:19.372252 7650 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5c8cdc07bf6f4480b269ca34a9b3e426" candidate_uuid: "81d8bacc3ca64d628a255ad416a77c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" I1121 10:21:19.372277 7650 raft_consensus.cc:2905] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.372319 7402 raft_consensus.cc:2359] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:19.372367 8022 raft_consensus.cc:2359] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:19.372465 7831 leader_election.cc:310] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d; no voters: I1121 10:21:19.372630 8503 raft_consensus.cc:2653] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:19.372684 8503 raft_consensus.cc:664] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Becoming Leader. State: Replica: 81d8bacc3ca64d628a255ad416a77c3d, State: Running, Role: LEADER I1121 10:21:19.372776 8503 consensus_queue.cc:208] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } I1121 10:21:19.373044 7650 raft_consensus.cc:2359] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81d8bacc3ca64d628a255ad416a77c3d in term 1. I1121 10:21:19.373147 8502 leader_election.cc:296] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [CANDIDATE]: Term 1 election: Requested vote from peers 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995), c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291), f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:19.373464 7271 catalog_manager.cc:4143] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: term changed from 0 to 1, leader changed from to 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5). New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } } I1121 10:21:19.392565 8307 raft_consensus.cc:469] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:19.392668 8307 raft_consensus.cc:491] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.392791 8307 leader_election.cc:296] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291) I1121 10:21:19.393049 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "f30bc779dfa840c6ae504b94d82eaa5a" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" is_pre_election: true I1121 10:21:19.393069 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "f30bc779dfa840c6ae504b94d82eaa5a" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" is_pre_election: true I1121 10:21:19.393097 7519 raft_consensus.cc:2359] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:19.393105 7893 raft_consensus.cc:2359] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:19.393117 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "f30bc779dfa840c6ae504b94d82eaa5a" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" is_pre_election: true I1121 10:21:19.393151 8022 raft_consensus.cc:2359] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:19.393214 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "f30bc779dfa840c6ae504b94d82eaa5a" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" is_pre_election: true I1121 10:21:19.393254 7773 raft_consensus.cc:2359] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 0. I1121 10:21:19.393280 7334 leader_election.cc:310] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 41118f5c2e8f48179ffbebbfb6da2edf, 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d; no voters: I1121 10:21:19.393343 8307 raft_consensus.cc:2653] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:19.393366 8307 raft_consensus.cc:469] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:19.393383 8307 raft_consensus.cc:2905] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.394188 8307 raft_consensus.cc:491] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.394287 8307 leader_election.cc:296] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Requested vote from peers c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833), 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445), 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993), 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291) I1121 10:21:19.394439 8022 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "f30bc779dfa840c6ae504b94d82eaa5a" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" I1121 10:21:19.394474 8022 raft_consensus.cc:2905] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.394536 7519 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "f30bc779dfa840c6ae504b94d82eaa5a" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" I1121 10:21:19.394544 7893 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "f30bc779dfa840c6ae504b94d82eaa5a" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d8bacc3ca64d628a255ad416a77c3d" I1121 10:21:19.394567 7893 raft_consensus.cc:2905] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.394569 7519 raft_consensus.cc:2905] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.395128 7893 raft_consensus.cc:2359] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:19.395200 8022 raft_consensus.cc:2359] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:19.395239 7519 raft_consensus.cc:2359] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:19.395321 7773 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "f30bc779dfa840c6ae504b94d82eaa5a" candidate_uuid: "63b9dc8cf49b47ebb644515af2f427e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3038ad6c564f438aa18383cddc769a" I1121 10:21:19.395344 7773 raft_consensus.cc:2905] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:19.395354 7333 leader_election.cc:310] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 63b9dc8cf49b47ebb644515af2f427e7, 81d8bacc3ca64d628a255ad416a77c3d, c9e9c35ff1cb4b68ba7cf5c037a98347; no voters: I1121 10:21:19.395494 8363 raft_consensus.cc:2653] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:19.395529 8363 raft_consensus.cc:664] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Becoming Leader. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Running, Role: LEADER I1121 10:21:19.395602 8363 consensus_queue.cc:208] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } I1121 10:21:19.396054 7773 raft_consensus.cc:2359] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63b9dc8cf49b47ebb644515af2f427e7 in term 1. I1121 10:21:19.396198 7271 catalog_manager.cc:4143] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: term changed from 0 to 1, leader changed from to 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } } W1121 10:21:19.469218 8542 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_4' I1121 10:21:19.498713 8022 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:19.498819 7650 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:19.498958 7402 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:19.499056 8502 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.499128 8327 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.499248 7519 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:19.499522 8502 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.499624 8327 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.500747 7773 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.501125 8023 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.501227 8021 raft_consensus.cc:1181] T 55d18b1e7ede4cb28f5dfc4ded166a30 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.501224 8286 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.501446 7773 raft_consensus.cc:1181] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.501520 8363 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.501598 7893 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.501670 8363 consensus_queue.cc:984] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.501821 7527 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.502920 8364 consensus_queue.cc:984] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.503131 8364 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.503602 8363 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.506400 7650 raft_consensus.cc:1181] T 55d18b1e7ede4cb28f5dfc4ded166a30 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.506901 7518 raft_consensus.cc:1181] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:19.508394 8557 consensus_queue.cc:984] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.513025 8558 consensus_queue.cc:984] T 55d18b1e7ede4cb28f5dfc4ded166a30 P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:19.513378 8334 mvcc.cc:203] Tried to move safe_time back from 6448462559228084224 to 6448462558804656128. Current Snapshot: MvccSnapshot[committed={T|T < 6448462559228084224 or (T in {6448462559228084224})}] I1121 10:21:19.513979 8413 mvcc.cc:203] Tried to move safe_time back from 6448462559228084224 to 6448462558804656128. Current Snapshot: MvccSnapshot[committed={T|T < 6448462559228084224 or (T in {6448462559228084224})}] I1121 10:21:19.515408 8402 mvcc.cc:203] Tried to move safe_time back from 6448462559231950848 to 6448462558385741824. Current Snapshot: MvccSnapshot[committed={T|T < 6448462559224684544}] I1121 10:21:19.516294 8331 mvcc.cc:203] Tried to move safe_time back from 6448462559228084224 to 6448462558804656128. Current Snapshot: MvccSnapshot[committed={T|T < 6448462559228084224 or (T in {6448462559228084224})}] Generator report time total : 31.8688 ms time per row: 0.0159344 ms I1121 10:21:19.519670 8346 mvcc.cc:203] Tried to move safe_time back from 6448462559222554624 to 6448462558804656128. Current Snapshot: MvccSnapshot[committed={T|T < 6448462559222554624}] I1121 10:21:19.521036 8343 mvcc.cc:203] Tried to move safe_time back from 6448462559227355136 to 6448462558711586816. Current Snapshot: MvccSnapshot[committed={T|T < 6448462559224856576}] I1121 10:21:19.524873 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.70.7:38147 --local_ip_for_outbound_sockets=127.5.70.7 --webserver_port=40471 --webserver_interface=127.5.70.7 --tserver_master_addrs=127.5.70.62:38839 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:19.562470 8582 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:19.562620 8582 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:19.563524 8582 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:19.563546 8582 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:19.563560 8582 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.7 I1121 10:21:19.564723 8582 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.7:38147 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.70.7 --webserver_port=40471 --tserver_master_addrs=127.5.70.62:38839 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.8582 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.7 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:19.566489 8582 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:19.567639 8582 hybrid_clock.cc:183] HybridClock initialized: now 1574331679567617 us; error 28 us; skew 500 ppm I1121 10:21:19.567840 8582 fs_manager.cc:261] Metadata directory not provided I1121 10:21:19.567895 8582 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:19.569211 8582 fs_manager.cc:404] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:21:19.569444 8582 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:19.569573 8582 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:19.570348 8582 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:19.570788 8582 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal uuid: "522e51b66e2c425091349ca2c9059244" format_stamp: "Formatted at 2019-11-21 10:21:14 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:19.570844 8582 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:19.620620 8582 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:19.621001 8582 ts_tablet_manager.cc:384] Loading tablet metadata (0/1 complete) I1121 10:21:19.621724 8582 ts_tablet_manager.cc:396] Loaded tablet metadata (1 total tablets, 1 live tablets) I1121 10:21:19.621749 8582 ts_tablet_manager.cc:402] Registering tablets (0/1 complete) I1121 10:21:19.622272 8582 ts_tablet_manager.cc:416] Registered 1 tablets I1121 10:21:19.622436 8599 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:19.631338 8582 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.7:38147 I1121 10:21:19.631800 8582 webserver.cc:382] Webserver started at http://127.5.70.7:40471/ using document root and password file I1121 10:21:19.632149 8582 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb I1121 10:21:19.636973 8704 heartbeater.cc:322] Connected to a master server at 127.5.70.62:38839 I1121 10:21:19.637037 8704 heartbeater.cc:406] Registering TS with master... I1121 10:21:19.637179 8704 heartbeater.cc:455] Master 127.5.70.62:38839 requested a full tablet report, sending... I1121 10:21:19.637562 8599 log.cc:810] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Log is configured to *not* fsync() on all Append() calls I1121 10:21:19.637665 7271 ts_manager.cc:192] Re-registered known tserver with Master: 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147) I1121 10:21:19.638211 7271 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.7:33269 I1121 10:21:19.638881 8599 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:19.639197 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 8582 I1121 10:21:19.639240 8599 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:19.639403 5400 test_util.cc:225] Using random seed: -1467611285 I1121 10:21:19.640154 8599 ts_tablet_manager.cc:1104] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s I1121 10:21:19.642580 8599 raft_consensus.cc:337] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.642658 8599 raft_consensus.cc:702] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: FOLLOWER I1121 10:21:19.642825 8599 consensus_queue.cc:231] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } I1121 10:21:19.642988 8704 heartbeater.cc:447] Master 127.5.70.62:38839 was elected leader, sending a full tablet report... I1121 10:21:19.643002 8599 ts_tablet_manager.cc:1132] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s I1121 10:21:19.644569 5400 test_workload.cc:329] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists I1121 10:21:19.644665 5400 test_util.cc:225] Using random seed: -1467606025 I1121 10:21:19.648995 5400 test_workload.cc:329] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists I1121 10:21:19.649077 5400 test_util.cc:225] Using random seed: -1467601612 I1121 10:21:19.659718 5400 test_workload.cc:329] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists I1121 10:21:19.659816 5400 test_util.cc:225] Using random seed: -1467590875 I1121 10:21:19.665729 5400 test_workload.cc:329] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists I1121 10:21:19.665818 5400 test_util.cc:225] Using random seed: -1467584872 I1121 10:21:19.672726 5400 test_workload.cc:329] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists W1121 10:21:19.882622 8705 tablet.cc:1767] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:19.986027 7896 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:19.987617 8020 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch) I1121 10:21:19.987884 8336 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I1121 10:21:19.989782 7773 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch) I1121 10:21:19.989893 7519 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch) I1121 10:21:19.990587 8336 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I1121 10:21:19.990777 8336 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I1121 10:21:19.993396 7650 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch) I1121 10:21:20.000607 8266 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index -1 to 11, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.000946 8336 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I1121 10:21:20.002226 7651 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index -1 to 11, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.002548 8020 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index -1 to 11, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.003960 7271 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index -1 to 11, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } W1121 10:21:20.005837 7832 consensus_peers.cc:473] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cd3c7862a7c420cb0b87bd87a770d84. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:20.006778 7519 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index -1 to 11, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.006969 7773 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index -1 to 11, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.016770 7402 consensus_queue.cc:208] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.019536 8020 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch) I1121 10:21:20.021955 7899 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 20. (index mismatch) I1121 10:21:20.023042 8569 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s I1121 10:21:20.023998 7519 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch) I1121 10:21:20.024430 7774 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch) I1121 10:21:20.026347 8569 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s I1121 10:21:20.027279 8569 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s I1121 10:21:20.027413 8569 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.000s I1121 10:21:20.032341 8558 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } W1121 10:21:20.037647 7334 consensus_peers.cc:473] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f30bc779dfa840c6ae504b94d82eaa5a. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:20.037855 7274 catalog_manager.cc:4143] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index -1 to 20, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:20.037900 7774 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.037973 7519 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.039204 8021 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.040861 7896 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.041754 7651 consensus_queue.cc:208] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 1.18, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.042366 8021 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch) I1121 10:21:20.042615 8400 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s I1121 10:21:20.043027 7390 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch) I1121 10:21:20.043495 8400 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s I1121 10:21:20.044204 8763 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.19: config changed from index -1 to 19, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.045948 7527 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch) I1121 10:21:20.047614 7390 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index -1 to 19, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.048182 8022 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index -1 to 19, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.048709 7774 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 18. Preceding OpId from leader: term: 1 index: 19. (index mismatch) I1121 10:21:20.049531 7271 catalog_manager.cc:4143] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index -1 to 19, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:20.049840 8764 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s I1121 10:21:20.049974 8764 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s I1121 10:21:20.050879 7774 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index -1 to 19, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.052673 7527 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.19: config changed from index -1 to 19, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.059226 7390 consensus_queue.cc:208] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.20, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.060431 8022 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch) W1121 10:21:20.061441 7584 consensus_peers.cc:473] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 993b9cf62f6c4e6296acbb32382ab037. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:20.061537 7650 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 20. Preceding OpId from leader: term: 1 index: 21. (index mismatch) I1121 10:21:20.062796 8561 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s I1121 10:21:20.062880 8568 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s I1121 10:21:20.064124 7775 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 21. (index mismatch) I1121 10:21:20.069005 7899 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 21. (index mismatch) I1121 10:21:20.069463 8561 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.21: config changed from index -1 to 21, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } W1121 10:21:20.069514 7334 consensus_peers.cc:473] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d59736e513f426095bedf84bbeec48d. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:20.074040 8560 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s I1121 10:21:20.074062 7650 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index -1 to 21, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.076777 8020 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index -1 to 21, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.078065 7274 catalog_manager.cc:4143] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index -1 to 21, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.080126 8560 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.000s I1121 10:21:20.081292 7519 consensus_queue.cc:208] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.081816 7650 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch) I1121 10:21:20.082015 8454 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s I1121 10:21:20.082146 7775 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch) I1121 10:21:20.082346 8454 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s I1121 10:21:20.082723 7402 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch) I1121 10:21:20.082945 8507 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index -1 to 33, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.083086 8505 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s I1121 10:21:20.083483 7402 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index -1 to 33, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.084275 7651 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index -1 to 33, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.084465 7772 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index -1 to 33, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.085722 7271 catalog_manager.cc:4143] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf reported cstate change: config changed from index -1 to 33, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:20.085824 7772 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index -1 to 21, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.090083 7896 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch) W1121 10:21:20.090374 7460 consensus_peers.cc:473] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 865e88694908474197879cd8d34789ad. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:20.091995 7896 consensus_queue.cc:208] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: true } } I1121 10:21:20.092852 7896 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.21: config changed from index -1 to 21, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.093652 8021 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch) I1121 10:21:20.094063 7402 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch) I1121 10:21:20.094103 8504 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s I1121 10:21:20.094533 8299 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s I1121 10:21:20.095849 8299 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: true } } } I1121 10:21:20.096652 8454 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s I1121 10:21:20.098821 7650 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch) I1121 10:21:20.099002 7772 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 1 index: 20. (index mismatch) I1121 10:21:20.099318 7402 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: true } } } I1121 10:21:20.099870 8020 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: true } } } I1121 10:21:20.100775 8299 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s I1121 10:21:20.101289 7274 catalog_manager.cc:4143] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index -1 to 20, NON_VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) added. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: true } } } I1121 10:21:20.102869 7896 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index -1 to 33, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:20.104933 8299 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s W1121 10:21:20.105072 7832 consensus_peers.cc:473] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291): Couldn't send request to peer 41118f5c2e8f48179ffbebbfb6da2edf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5abf7a6c891418482e200e7115e2c2d. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:20.105944 7772 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: true } } } I1121 10:21:20.107185 7650 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.20: config changed from index -1 to 20, NON_VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) added. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: true } } } I1121 10:21:20.492071 8960 ts_tablet_manager.cc:721] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995) I1121 10:21:20.492154 8960 tablet_copy_client.cc:243] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:36995 I1121 10:21:20.502275 8962 ts_tablet_manager.cc:721] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995) I1121 10:21:20.502859 8962 tablet_copy_client.cc:243] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:36995 I1121 10:21:20.506428 7422 tablet_copy_service.cc:135] P 63b9dc8cf49b47ebb644515af2f427e7: Received BeginTabletCopySession request for tablet 7d59736e513f426095bedf84bbeec48d from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:36379) I1121 10:21:20.506470 7422 tablet_copy_service.cc:156] P 63b9dc8cf49b47ebb644515af2f427e7: Beginning new tablet copy session on tablet 7d59736e513f426095bedf84bbeec48d from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:36379: session id = 522e51b66e2c425091349ca2c9059244-7d59736e513f426095bedf84bbeec48d I1121 10:21:20.507335 7422 tablet_copy_source_session.cc:204] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:20.509366 7422 tablet_copy_service.cc:135] P 63b9dc8cf49b47ebb644515af2f427e7: Received BeginTabletCopySession request for tablet f30bc779dfa840c6ae504b94d82eaa5a from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:36379) I1121 10:21:20.509390 7422 tablet_copy_service.cc:156] P 63b9dc8cf49b47ebb644515af2f427e7: Beginning new tablet copy session on tablet f30bc779dfa840c6ae504b94d82eaa5a from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:36379: session id = 522e51b66e2c425091349ca2c9059244-f30bc779dfa840c6ae504b94d82eaa5a I1121 10:21:20.509876 7422 tablet_copy_source_session.cc:204] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:20.511445 8960 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7d59736e513f426095bedf84bbeec48d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:20.512617 8962 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet f30bc779dfa840c6ae504b94d82eaa5a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:20.516629 8960 tablet_copy_client.cc:581] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:20.516868 8960 tablet_copy_client.cc:544] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:20.516722 8962 tablet_copy_client.cc:581] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:20.517258 8962 tablet_copy_client.cc:544] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:20.523010 8962 tablet_copy_client.cc:420] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:20.527485 8960 tablet_copy_client.cc:420] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:20.525722 8963 ts_tablet_manager.cc:721] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291) I1121 10:21:20.529239 8963 tablet_copy_client.cc:243] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:39291 I1121 10:21:20.533478 7547 tablet_copy_service.cc:135] P 41118f5c2e8f48179ffbebbfb6da2edf: Received BeginTabletCopySession request for tablet 865e88694908474197879cd8d34789ad from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:48463) I1121 10:21:20.533514 7547 tablet_copy_service.cc:156] P 41118f5c2e8f48179ffbebbfb6da2edf: Beginning new tablet copy session on tablet 865e88694908474197879cd8d34789ad from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:48463: session id = 522e51b66e2c425091349ca2c9059244-865e88694908474197879cd8d34789ad I1121 10:21:20.533746 8962 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:20.534155 7547 tablet_copy_source_session.cc:204] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:20.535673 8960 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:20.535799 8963 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 865e88694908474197879cd8d34789ad. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:20.547322 8962 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=41 ignored=0} inserts{seen=713 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I1121 10:21:20.547682 8962 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:20.547940 8962 ts_tablet_manager.cc:1104] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s I1121 10:21:20.548171 8962 raft_consensus.cc:337] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.548931 8962 raft_consensus.cc:702] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:20.549139 8962 consensus_queue.cc:231] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 1.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.549965 8962 ts_tablet_manager.cc:1132] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:20.551239 7422 tablet_copy_service.cc:325] P 63b9dc8cf49b47ebb644515af2f427e7: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-f30bc779dfa840c6ae504b94d82eaa5a received from {username='slave'} at 127.5.70.7:36379 I1121 10:21:20.551280 7422 tablet_copy_service.cc:413] P 63b9dc8cf49b47ebb644515af2f427e7: ending tablet copy session 522e51b66e2c425091349ca2c9059244-f30bc779dfa840c6ae504b94d82eaa5a on tablet f30bc779dfa840c6ae504b94d82eaa5a with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:20.555173 8963 tablet_copy_client.cc:581] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:20.555922 8963 tablet_copy_client.cc:544] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:20.559746 8963 tablet_copy_client.cc:420] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:20.564182 8963 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:20.571485 8969 ts_tablet_manager.cc:721] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: Initiating tablet copy from peer 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:20.571554 8969 tablet_copy_client.cc:243] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.5:36445 I1121 10:21:20.572815 7919 tablet_copy_service.cc:135] P 81d8bacc3ca64d628a255ad416a77c3d: Received BeginTabletCopySession request for tablet c5abf7a6c891418482e200e7115e2c2d from peer 41118f5c2e8f48179ffbebbfb6da2edf ({username='slave'} at 127.5.70.2:55933) I1121 10:21:20.572851 7919 tablet_copy_service.cc:156] P 81d8bacc3ca64d628a255ad416a77c3d: Beginning new tablet copy session on tablet c5abf7a6c891418482e200e7115e2c2d from peer 41118f5c2e8f48179ffbebbfb6da2edf at {username='slave'} at 127.5.70.2:55933: session id = 41118f5c2e8f48179ffbebbfb6da2edf-c5abf7a6c891418482e200e7115e2c2d I1121 10:21:20.573621 7919 tablet_copy_source_session.cc:204] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:20.574769 8969 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c5abf7a6c891418482e200e7115e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:20.583158 8969 tablet_copy_client.cc:581] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: tablet copy: Starting download of 0 data blocks... I1121 10:21:20.583369 8969 tablet_copy_client.cc:544] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: tablet copy: Starting download of 1 WAL segments... I1121 10:21:20.585031 8969 tablet_copy_client.cc:420] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:20.586026 8969 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap starting. I1121 10:21:20.594558 8960 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=42 ignored=0} inserts{seen=739 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:20.595104 8960 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:20.595266 8960 ts_tablet_manager.cc:1104] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.004s I1121 10:21:20.595546 8960 raft_consensus.cc:337] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.596568 8960 raft_consensus.cc:702] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:20.596694 8960 consensus_queue.cc:231] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 42, Last appended: 1.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.597702 8960 ts_tablet_manager.cc:1132] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1121 10:21:20.600281 7422 tablet_copy_service.cc:325] P 63b9dc8cf49b47ebb644515af2f427e7: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-7d59736e513f426095bedf84bbeec48d received from {username='slave'} at 127.5.70.7:36379 I1121 10:21:20.600306 7422 tablet_copy_service.cc:413] P 63b9dc8cf49b47ebb644515af2f427e7: ending tablet copy session 522e51b66e2c425091349ca2c9059244-7d59736e513f426095bedf84bbeec48d on tablet 7d59736e513f426095bedf84bbeec48d with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:20.601665 8963 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=58 overwritten=0 applied=58 ignored=0} inserts{seen=730 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:20.602039 8963 tablet_bootstrap.cc:439] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:20.602180 8963 ts_tablet_manager.cc:1104] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.007s I1121 10:21:20.602370 8963 raft_consensus.cc:337] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.602458 8963 raft_consensus.cc:702] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:20.602567 8963 consensus_queue.cc:231] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 58, Last appended: 1.58, Last appended by leader: 58, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.603271 8963 ts_tablet_manager.cc:1132] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:20.603592 7547 tablet_copy_service.cc:325] P 41118f5c2e8f48179ffbebbfb6da2edf: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-865e88694908474197879cd8d34789ad received from {username='slave'} at 127.5.70.7:48463 I1121 10:21:20.603622 7547 tablet_copy_service.cc:413] P 41118f5c2e8f48179ffbebbfb6da2edf: ending tablet copy session 522e51b66e2c425091349ca2c9059244-865e88694908474197879cd8d34789ad on tablet 865e88694908474197879cd8d34789ad with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:20.608768 8963 ts_tablet_manager.cc:721] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:20.608844 8963 tablet_copy_client.cc:243] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:43909 I1121 10:21:20.612144 8960 ts_tablet_manager.cc:721] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:20.612234 8960 tablet_copy_client.cc:243] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.5:36445 I1121 10:21:20.612251 7671 tablet_copy_service.cc:135] P f5bc6d2f9c574b4db9396f9f900c1141: Received BeginTabletCopySession request for tablet 993b9cf62f6c4e6296acbb32382ab037 from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:43955) I1121 10:21:20.612277 7671 tablet_copy_service.cc:156] P f5bc6d2f9c574b4db9396f9f900c1141: Beginning new tablet copy session on tablet 993b9cf62f6c4e6296acbb32382ab037 from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:43955: session id = 522e51b66e2c425091349ca2c9059244-993b9cf62f6c4e6296acbb32382ab037 I1121 10:21:20.613234 7671 tablet_copy_source_session.cc:204] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:20.613688 8963 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 993b9cf62f6c4e6296acbb32382ab037. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:20.614979 8963 tablet_copy_client.cc:581] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:20.615140 8963 tablet_copy_client.cc:544] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:20.615778 7919 tablet_copy_service.cc:135] P 81d8bacc3ca64d628a255ad416a77c3d: Received BeginTabletCopySession request for tablet 7cd3c7862a7c420cb0b87bd87a770d84 from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:53339) I1121 10:21:20.615813 7919 tablet_copy_service.cc:156] P 81d8bacc3ca64d628a255ad416a77c3d: Beginning new tablet copy session on tablet 7cd3c7862a7c420cb0b87bd87a770d84 from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:53339: session id = 522e51b66e2c425091349ca2c9059244-7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:20.616092 8963 tablet_copy_client.cc:420] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:20.616359 7919 tablet_copy_source_session.cc:204] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:20.616863 8960 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7cd3c7862a7c420cb0b87bd87a770d84. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:20.617341 8963 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:20.618166 8960 tablet_copy_client.cc:581] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:20.622309 8960 tablet_copy_client.cc:544] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:20.623180 8960 tablet_copy_client.cc:420] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:20.624130 8960 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:20.648248 8969 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap replayed 1/1 log segments. Stats: ops{read=41 overwritten=0 applied=41 ignored=0} inserts{seen=660 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:20.648811 8969 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: Bootstrap complete. I1121 10:21:20.648994 8969 ts_tablet_manager.cc:1104] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.004s I1121 10:21:20.649276 8969 raft_consensus.cc:337] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: true } } I1121 10:21:20.649379 8969 raft_consensus.cc:702] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41118f5c2e8f48179ffbebbfb6da2edf, State: Initialized, Role: LEARNER I1121 10:21:20.649492 8969 consensus_queue.cc:231] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 1.41, Last appended by leader: 41, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: true } } I1121 10:21:20.650292 8969 ts_tablet_manager.cc:1132] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:20.650526 7919 tablet_copy_service.cc:325] P 81d8bacc3ca64d628a255ad416a77c3d: Request end of tablet copy session 41118f5c2e8f48179ffbebbfb6da2edf-c5abf7a6c891418482e200e7115e2c2d received from {username='slave'} at 127.5.70.2:55933 I1121 10:21:20.650550 7919 tablet_copy_service.cc:413] P 81d8bacc3ca64d628a255ad416a77c3d: ending tablet copy session 41118f5c2e8f48179ffbebbfb6da2edf-c5abf7a6c891418482e200e7115e2c2d on tablet c5abf7a6c891418482e200e7115e2c2d with peer 41118f5c2e8f48179ffbebbfb6da2edf I1121 10:21:20.673699 8960 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=50 overwritten=0 applied=50 ignored=0} inserts{seen=747 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:20.675483 8960 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:20.675655 8960 ts_tablet_manager.cc:1104] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.008s I1121 10:21:20.676532 8960 raft_consensus.cc:337] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.676637 8960 raft_consensus.cc:702] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:20.676751 8960 consensus_queue.cc:231] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 50, Last appended: 1.50, Last appended by leader: 50, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.677881 8960 ts_tablet_manager.cc:1132] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1121 10:21:20.678269 7919 tablet_copy_service.cc:325] P 81d8bacc3ca64d628a255ad416a77c3d: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-7cd3c7862a7c420cb0b87bd87a770d84 received from {username='slave'} at 127.5.70.7:53339 I1121 10:21:20.678297 7919 tablet_copy_service.cc:413] P 81d8bacc3ca64d628a255ad416a77c3d: ending tablet copy session 522e51b66e2c425091349ca2c9059244-7cd3c7862a7c420cb0b87bd87a770d84 on tablet 7cd3c7862a7c420cb0b87bd87a770d84 with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:20.682293 8963 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=56 overwritten=0 applied=56 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:20.682613 8963 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:20.682775 8963 ts_tablet_manager.cc:1104] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.006s I1121 10:21:20.682984 8963 raft_consensus.cc:337] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.683075 8963 raft_consensus.cc:702] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:20.683187 8963 consensus_queue.cc:231] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:20.684087 8963 ts_tablet_manager.cc:1132] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:20.684381 7671 tablet_copy_service.cc:325] P f5bc6d2f9c574b4db9396f9f900c1141: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-993b9cf62f6c4e6296acbb32382ab037 received from {username='slave'} at 127.5.70.7:43955 I1121 10:21:20.684409 7671 tablet_copy_service.cc:413] P f5bc6d2f9c574b4db9396f9f900c1141: ending tablet copy session 522e51b66e2c425091349ca2c9059244-993b9cf62f6c4e6296acbb32382ab037 on tablet 993b9cf62f6c4e6296acbb32382ab037 with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:20.939477 8661 raft_consensus.cc:1123] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.32->[1.33-1.90] Dedup: 1.58->[1.59-1.90] I1121 10:21:20.957402 8268 raft_consensus.cc:971] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:20.957628 8268 consensus_queue.cc:208] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:20.958178 7402 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch) I1121 10:21:20.958492 8455 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s I1121 10:21:20.958652 7649 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch) I1121 10:21:20.958889 8455 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s I1121 10:21:20.959827 7772 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch) I1121 10:21:20.959869 8661 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch) I1121 10:21:20.959872 7893 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch) I1121 10:21:20.960337 8454 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s I1121 10:21:20.960422 8454 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s I1121 10:21:20.960889 8454 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s I1121 10:21:20.961777 8506 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 33 to 93, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.962040 7772 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 33 to 93, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.962152 8661 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 33 to 93, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.963209 7899 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 33 to 93, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.963313 7649 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 33 to 93, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.963359 7271 catalog_manager.cc:4143] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a reported cstate change: config changed from index 33 to 93, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.964495 7402 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 33 to 93, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.969854 7525 consensus_queue.cc:208] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:20.970340 7648 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch) I1121 10:21:20.970525 8501 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s I1121 10:21:20.970631 7402 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch) I1121 10:21:20.970854 8455 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s I1121 10:21:20.971268 8661 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch) I1121 10:21:20.971415 8455 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index 93 to 94, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.971565 8501 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s I1121 10:21:20.971879 8661 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 93 to 94, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.971941 7773 raft_consensus.cc:1181] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 41118f5c2e8f48179ffbebbfb6da2edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch) I1121 10:21:20.973117 7273 catalog_manager.cc:4143] T 865e88694908474197879cd8d34789ad P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 93 to 94, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New cstate: current_term: 1 leader_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.973771 7649 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 93 to 94, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.973807 8454 consensus_queue.cc:984] T 865e88694908474197879cd8d34789ad P 41118f5c2e8f48179ffbebbfb6da2edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s I1121 10:21:20.973975 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 865e88694908474197879cd8d34789ad with cas_config_opid_index 93: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:20.976044 7402 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 93 to 94, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.976104 7772 raft_consensus.cc:2801] T 865e88694908474197879cd8d34789ad P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index 93 to 94, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:20.977530 7874 tablet_service.cc:1067] Processing DeleteTablet for tablet 865e88694908474197879cd8d34789ad with delete_type TABLET_DATA_TOMBSTONED (TS 81d8bacc3ca64d628a255ad416a77c3d not found in new config with opid_index 94) from {username='slave'} at 127.0.0.1:37730 I1121 10:21:20.978657 9032 tablet_replica.cc:282] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d: stopping tablet replica I1121 10:21:20.985142 9032 raft_consensus.cc:2156] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:20.985446 9032 raft_consensus.cc:2183] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:20.988739 9032 ts_tablet_manager.cc:1457] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:21.000982 9032 ts_tablet_manager.cc:1470] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.93 I1121 10:21:21.005373 9032 log.cc:1153] T 865e88694908474197879cd8d34789ad P 81d8bacc3ca64d628a255ad416a77c3d: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal/wals/865e88694908474197879cd8d34789ad I1121 10:21:21.008592 7263 catalog_manager.cc:3512] TS 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445): tablet 865e88694908474197879cd8d34789ad (table rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]) successfully deleted I1121 10:21:21.042527 8661 raft_consensus.cc:1123] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.20->[1.21-1.64] Dedup: 1.42->[1.43-1.64] I1121 10:21:21.047076 7525 raft_consensus.cc:1123] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.19->[1.20-1.59] Dedup: 1.41->[1.42-1.59] I1121 10:21:21.055359 8970 raft_consensus.cc:971] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:21.055830 8970 consensus_queue.cc:208] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:21.057476 8660 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch) I1121 10:21:21.057670 8021 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch) I1121 10:21:21.058168 7899 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch) I1121 10:21:21.058714 8569 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s I1121 10:21:21.058825 8562 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s I1121 10:21:21.059295 7650 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch) I1121 10:21:21.059379 8562 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s I1121 10:21:21.063017 8867 raft_consensus.cc:971] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d: attempting to promote NON_VOTER 41118f5c2e8f48179ffbebbfb6da2edf to VOTER I1121 10:21:21.063220 7771 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch) I1121 10:21:21.063228 8867 consensus_queue.cc:208] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.59, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } I1121 10:21:21.063485 8569 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s I1121 10:21:21.063822 8021 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch) I1121 10:21:21.064167 7402 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch) I1121 10:21:21.064235 8327 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s I1121 10:21:21.064380 8327 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s I1121 10:21:21.065158 7525 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 LEARNER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch) I1121 10:21:21.065397 8502 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s I1121 10:21:21.065423 7771 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch) I1121 10:21:21.066398 8562 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index 21 to 66, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.066704 8327 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.60: config changed from index 20 to 60, 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.067770 7651 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 59. Preceding OpId from leader: term: 1 index: 60. (index mismatch) I1121 10:21:21.067901 8327 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s I1121 10:21:21.067932 8569 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 64, Time since last communication: 0.000s I1121 10:21:21.068099 8021 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 20 to 60, 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.069031 7278 catalog_manager.cc:4143] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 21 to 66, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:21.069435 7399 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 20 to 60, 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.069659 7275 catalog_manager.cc:4143] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index 20 to 60, 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.070219 8021 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 21 to 66, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.070741 8661 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 21 to 66, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.071527 7899 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 21 to 66, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.073037 8502 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 59, Time since last communication: 0.000s I1121 10:21:21.073232 7525 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 20 to 60, 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.073482 7650 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 20 to 60, 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.076061 7771 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.60: config changed from index 20 to 60, 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 60 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.077028 7650 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 21 to 66, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.077433 7771 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index 21 to 66, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.084134 7399 consensus_queue.cc:208] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 1.67, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:21.084887 8661 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch) I1121 10:21:21.085039 7893 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch) I1121 10:21:21.085067 8568 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I1121 10:21:21.085484 8568 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I1121 10:21:21.086419 8569 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.68: config changed from index 66 to 68, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.086758 8661 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 66 to 68, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.087705 8019 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch) I1121 10:21:21.087837 7278 catalog_manager.cc:4143] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 66 to 68, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.088038 7262 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 7d59736e513f426095bedf84bbeec48d with cas_config_opid_index 66: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:21.088457 7893 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 66 to 68, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.092036 7899 consensus_queue.cc:208] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.61, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } I1121 10:21:21.092555 8019 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch) I1121 10:21:21.092555 7399 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch) I1121 10:21:21.092681 8562 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I1121 10:21:21.092785 7650 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch) I1121 10:21:21.092900 8327 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s I1121 10:21:21.093109 8503 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s I1121 10:21:21.093698 8502 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 61, Time since last communication: 0.000s I1121 10:21:21.094382 8867 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.62: config changed from index 60 to 62, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.094383 7771 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 67. Preceding OpId from leader: term: 1 index: 68. (index mismatch) I1121 10:21:21.094470 7628 tablet_service.cc:1067] Processing DeleteTablet for tablet 7d59736e513f426095bedf84bbeec48d with delete_type TABLET_DATA_TOMBSTONED (TS f5bc6d2f9c574b4db9396f9f900c1141 not found in new config with opid_index 68) from {username='slave'} at 127.0.0.1:39024 I1121 10:21:21.094625 9051 tablet_replica.cc:282] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141: stopping tablet replica I1121 10:21:21.094806 9051 raft_consensus.cc:2156] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:21.094907 9051 raft_consensus.cc:2183] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:21.095557 8021 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 60 to 62, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.096786 7273 catalog_manager.cc:4143] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index 60 to 62, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.097167 9051 ts_tablet_manager.cc:1457] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:21.097854 7754 tablet_service.cc:1067] Processing DeleteTablet for tablet c5abf7a6c891418482e200e7115e2c2d with delete_type TABLET_DATA_TOMBSTONED (TS 1b3038ad6c564f438aa18383cddc769a not found in new config with opid_index 62) from {username='slave'} at 127.0.0.1:42904 I1121 10:21:21.099335 7527 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 61. Preceding OpId from leader: term: 1 index: 62. (index mismatch) I1121 10:21:21.099376 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet c5abf7a6c891418482e200e7115e2c2d with cas_config_opid_index 60: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:21.099977 9051 ts_tablet_manager.cc:1470] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.67 I1121 10:21:21.100020 9051 log.cc:1153] T 7d59736e513f426095bedf84bbeec48d P f5bc6d2f9c574b4db9396f9f900c1141: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/7d59736e513f426095bedf84bbeec48d I1121 10:21:21.100024 8661 raft_consensus.cc:1123] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.83] Dedup: 1.50->[1.51-1.83] I1121 10:21:21.102133 8327 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 60, Time since last communication: 0.000s I1121 10:21:21.102555 7527 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 60 to 62, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.104732 7399 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 60 to 62, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.106400 7650 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.62: config changed from index 60 to 62, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 62 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } } I1121 10:21:21.107400 9052 tablet_replica.cc:282] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: stopping tablet replica I1121 10:21:21.107498 9052 raft_consensus.cc:2156] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:21.107532 9052 pending_rounds.cc:64] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: Trying to abort 1 pending transactions. I1121 10:21:21.107643 9052 pending_rounds.cc:70] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: Aborting transaction as it isn't in flight: id { term: 1 index: 61 } timestamp: 6448462565711630336 op_type: WRITE_OP write_request { tablet_id: "c5abf7a6c891418482e200e7115e2c2d" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\024\366#/k\225\206\201\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "8299240f281cac3c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6448462565662171136 authz_token { token_data: "\010\313\314\331\356\005\"3\n\005slave\022*\n 1b28b579b64d471781f82b9f40e804ca\020\001\030\001 \001(\001" signature: "n?4H\251`\264\354\203C!!\027\031hzEK~`\001\352-\027\311\272\357\240f\275\254\334\323\021\377\030)\245\323\277\313\220\201t\356\t\247P\356\n0D\266\027\266\000\204\264\3744\370\353\2102" signing_key_seq_num: 0 } } request_id { client_id: "da8a7ba9717e4f52bcf91a68d1651206" seq_no: 188 first_incomplete_seq_no: 188 attempt_no: 0 } I1121 10:21:21.107728 9052 raft_consensus.cc:2183] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:21.107878 7261 catalog_manager.cc:3512] TS f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909): tablet 7d59736e513f426095bedf84bbeec48d (table rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]) successfully deleted I1121 10:21:21.108043 8021 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 66 to 68, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.108294 9052 ts_tablet_manager.cc:1457] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:21.110494 9052 ts_tablet_manager.cc:1470] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.61 I1121 10:21:21.110543 9052 log.cc:1153] T c5abf7a6c891418482e200e7115e2c2d P 1b3038ad6c564f438aa18383cddc769a: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/c5abf7a6c891418482e200e7115e2c2d I1121 10:21:21.110996 7263 catalog_manager.cc:3512] TS 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993): tablet c5abf7a6c891418482e200e7115e2c2d (table rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]) successfully deleted I1121 10:21:21.112025 8561 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s I1121 10:21:21.118505 8661 raft_consensus.cc:1123] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.19->[1.20-1.69] Dedup: 1.42->[1.43-1.69] I1121 10:21:21.123490 7772 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.68: config changed from index 66 to 68, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 68 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.130656 8562 raft_consensus.cc:971] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:21.131111 8562 consensus_queue.cc:208] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:21.131989 7899 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch) I1121 10:21:21.131994 7525 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch) I1121 10:21:21.132114 8660 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch) I1121 10:21:21.132174 8561 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s I1121 10:21:21.132912 8561 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s I1121 10:21:21.133185 8561 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s I1121 10:21:21.136629 8661 raft_consensus.cc:1123] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.18->[1.19-1.87] Dedup: 1.56->[1.57-1.87] I1121 10:21:21.141337 8570 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index 20 to 71, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.144585 7273 catalog_manager.cc:4143] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 20 to 71, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:21.144670 8660 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 20 to 71, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.145416 7899 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 20 to 71, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.145494 7772 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch) I1121 10:21:21.146391 8019 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 70. Preceding OpId from leader: term: 1 index: 71. (index mismatch) I1121 10:21:21.146610 8561 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s I1121 10:21:21.146690 7525 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 20 to 71, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.149394 8561 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 70, Time since last communication: 0.000s I1121 10:21:21.151604 7772 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 20 to 71, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.152876 8021 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index 20 to 71, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.157490 8336 raft_consensus.cc:971] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:21.157703 8336 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:21.158666 8021 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch) I1121 10:21:21.158848 8661 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch) I1121 10:21:21.158948 8327 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s I1121 10:21:21.159128 7771 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch) I1121 10:21:21.159445 7527 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch) I1121 10:21:21.159448 8502 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s I1121 10:21:21.159608 8327 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s I1121 10:21:21.159771 8327 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s I1121 10:21:21.160773 8502 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index 11 to 85, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.161048 7650 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch) I1121 10:21:21.162600 8266 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s I1121 10:21:21.163383 8661 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 11 to 85, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.163499 7275 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index 11 to 85, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:21.164585 8021 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 11 to 85, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.165047 7650 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 11 to 85, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.165640 7527 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 11 to 85, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.167959 7771 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 11 to 85, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.173866 7899 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:21.174521 8659 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch) I1121 10:21:21.174937 8504 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s I1121 10:21:21.175120 7390 consensus_queue.cc:208] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:21.177664 7773 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch) I1121 10:21:21.178033 7650 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch) I1121 10:21:21.179419 8021 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch) I1121 10:21:21.179775 8266 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s I1121 10:21:21.180244 7771 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch) I1121 10:21:21.181113 8660 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch) I1121 10:21:21.181265 7519 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch) I1121 10:21:21.181542 8266 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s I1121 10:21:21.181565 7899 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch) I1121 10:21:21.182274 8560 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 71, Time since last communication: 0.000s I1121 10:21:21.183224 8562 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 71, Time since last communication: 0.000s I1121 10:21:21.184011 8504 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s I1121 10:21:21.184476 8563 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 71, Time since last communication: 0.000s I1121 10:21:21.185094 8336 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index 85 to 87, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.185585 8563 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 71, Time since last communication: 0.000s I1121 10:21:21.186568 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 7cd3c7862a7c420cb0b87bd87a770d84 with cas_config_opid_index 85: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:21.186664 7651 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 85 to 87, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.186800 7275 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index 85 to 87, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:21.187863 8021 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 85 to 87, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.188093 7771 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 85 to 87, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.188719 8661 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 85 to 87, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.189812 7501 tablet_service.cc:1067] Processing DeleteTablet for tablet 7cd3c7862a7c420cb0b87bd87a770d84 with delete_type TABLET_DATA_TOMBSTONED (TS 41118f5c2e8f48179ffbebbfb6da2edf not found in new config with opid_index 87) from {username='slave'} at 127.0.0.1:49436 I1121 10:21:21.190217 9057 tablet_replica.cc:282] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf: stopping tablet replica I1121 10:21:21.190330 9057 raft_consensus.cc:2156] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:21.190377 9057 raft_consensus.cc:2183] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:21.190917 9057 ts_tablet_manager.cc:1457] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:21.192579 9057 ts_tablet_manager.cc:1470] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.86 I1121 10:21:21.192631 9057 log.cc:1153] T 7cd3c7862a7c420cb0b87bd87a770d84 P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:21.193091 8560 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 71 to 73, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.193104 7263 catalog_manager.cc:3512] TS 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291): tablet 7cd3c7862a7c420cb0b87bd87a770d84 (table rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]) successfully deleted I1121 10:21:21.194257 7262 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet f30bc779dfa840c6ae504b94d82eaa5a with cas_config_opid_index 71: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:21.195117 8763 raft_consensus.cc:971] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:21.195339 8763 consensus_queue.cc:208] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 87, Committed index: 87, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:21.196198 7273 catalog_manager.cc:4143] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 71 to 73, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:21.196979 8658 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch) I1121 10:21:21.197002 7519 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch) I1121 10:21:21.197252 7772 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 71 to 73, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.197464 8660 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 71 to 73, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.197572 7390 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch) I1121 10:21:21.197587 7773 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch) I1121 10:21:21.197747 8003 tablet_service.cc:1067] Processing DeleteTablet for tablet f30bc779dfa840c6ae504b94d82eaa5a with delete_type TABLET_DATA_TOMBSTONED (TS c9e9c35ff1cb4b68ba7cf5c037a98347 not found in new config with opid_index 73) from {username='slave'} at 127.0.0.1:44220 I1121 10:21:21.198087 8019 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch) I1121 10:21:21.198101 8762 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 87, Time since last communication: 0.000s I1121 10:21:21.198406 7519 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 71 to 73, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.199393 8273 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 87, Time since last communication: 0.000s I1121 10:21:21.199484 8273 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 87, Time since last communication: 0.000s I1121 10:21:21.200258 9058 tablet_replica.cc:282] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: stopping tablet replica I1121 10:21:21.200377 9058 raft_consensus.cc:2156] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:21.200445 9058 pending_rounds.cc:64] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: Trying to abort 1 pending transactions. I1121 10:21:21.200467 8762 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 87, Time since last communication: 0.000s I1121 10:21:21.201018 9058 pending_rounds.cc:70] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: Aborting transaction as it isn't in flight: id { term: 1 index: 72 } timestamp: 6448462566083530752 op_type: WRITE_OP write_request { tablet_id: "f30bc779dfa840c6ae504b94d82eaa5a" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000:/X\"\300\032\270\356\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "ae3880707ef0a189" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6448462566042087424 authz_token { token_data: "\010\313\314\331\356\005\"3\n\005slave\022*\n 692cf724e38648ed81e17afafb0bbf35\020\001\030\001 \001(\001" signature: "\267\340(\312j\024+\313{rGp\320\271\346\201,!\203+ :\363N\223`D=Q\227\'\221\337H\310\2524\215\031.G\343N\206\217\261P\377`\357\t\006\002\204+Pz\333\245\025t\025Xn" signing_key_seq_num: 0 } } request_id { client_id: "93b2303c34784b7d88db2a408f059a79" seq_no: 238 first_incomplete_seq_no: 238 attempt_no: 0 } I1121 10:21:21.201220 9058 raft_consensus.cc:2183] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:21.201454 8762 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 87, Time since last communication: 0.000s I1121 10:21:21.202467 7899 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 71 to 73, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.202606 9058 ts_tablet_manager.cc:1457] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:21.203570 8764 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index 19 to 89, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.204605 9058 ts_tablet_manager.cc:1470] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.72 I1121 10:21:21.204735 9058 log.cc:1153] T f30bc779dfa840c6ae504b94d82eaa5a P c9e9c35ff1cb4b68ba7cf5c037a98347: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal/wals/f30bc779dfa840c6ae504b94d82eaa5a I1121 10:21:21.205574 7519 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 19 to 89, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.205662 7273 catalog_manager.cc:4143] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index 19 to 89, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:21.206070 8661 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 19 to 89, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.208998 7390 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 19 to 89, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.210641 7772 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 19 to 89, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.212772 7262 catalog_manager.cc:3512] TS c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833): tablet f30bc779dfa840c6ae504b94d82eaa5a (table rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]) successfully deleted I1121 10:21:21.212954 8019 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index 19 to 89, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.216786 7650 consensus_queue.cc:208] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:21.218819 7772 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch) I1121 10:21:21.219589 7527 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch) I1121 10:21:21.219863 8660 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch) I1121 10:21:21.220479 8019 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch) I1121 10:21:21.220655 8347 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s I1121 10:21:21.220794 8400 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s I1121 10:21:21.220877 8920 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s I1121 10:21:21.223943 8400 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index 89 to 91, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.224591 8273 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s I1121 10:21:21.224709 8660 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 89 to 91, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.225299 7527 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 89 to 91, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.226186 7275 catalog_manager.cc:4143] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 89 to 91, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.227866 7771 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 89 to 91, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.228302 7370 tablet_service.cc:1067] Processing DeleteTablet for tablet 993b9cf62f6c4e6296acbb32382ab037 with delete_type TABLET_DATA_TOMBSTONED (TS 63b9dc8cf49b47ebb644515af2f427e7 not found in new config with opid_index 91) from {username='slave'} at 127.0.0.1:34708 I1121 10:21:21.229663 7261 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 993b9cf62f6c4e6296acbb32382ab037 with cas_config_opid_index 89: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:21.230370 8019 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 89 to 91, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:21.232573 9069 tablet_replica.cc:282] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7: stopping tablet replica I1121 10:21:21.232664 9069 raft_consensus.cc:2156] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:21.232707 9069 raft_consensus.cc:2183] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:21.234748 9069 ts_tablet_manager.cc:1457] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:21.236587 9069 ts_tablet_manager.cc:1470] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.90 I1121 10:21:21.236634 9069 log.cc:1153] T 993b9cf62f6c4e6296acbb32382ab037 P 63b9dc8cf49b47ebb644515af2f427e7: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/993b9cf62f6c4e6296acbb32382ab037 I1121 10:21:21.237151 7262 catalog_manager.cc:3512] TS 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995): tablet 993b9cf62f6c4e6296acbb32382ab037 (table rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]) successfully deleted I1121 10:21:21.327062 7650 consensus_queue.cc:208] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.328673 7893 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch) I1121 10:21:21.328685 7774 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch) I1121 10:21:21.329059 7399 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch) I1121 10:21:21.329339 8920 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s I1121 10:21:21.329624 8920 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s I1121 10:21:21.330276 7518 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch) W1121 10:21:21.330674 7584 consensus_peers.cc:473] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 787130b10eda4a71b2212bdb7ed2f140. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:21.330809 8273 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s I1121 10:21:21.331092 8920 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.331126 7893 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.331328 7518 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.332849 7273 catalog_manager.cc:4143] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf reported cstate change: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.333427 7771 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.337867 8764 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s I1121 10:21:21.343173 7399 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.346895 7649 consensus_queue.cc:208] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.350262 7399 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch) I1121 10:21:21.350385 7774 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch) I1121 10:21:21.350543 7525 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch) I1121 10:21:21.350667 8273 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s W1121 10:21:21.351240 7584 consensus_peers.cc:473] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d84faff8cd0f433d8de8664fd0054d2d. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:21.351325 8273 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s I1121 10:21:21.352797 8764 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s I1121 10:21:21.353030 8019 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch) I1121 10:21:21.353808 8347 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index -1 to 110, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.354557 7525 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index -1 to 110, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.355921 7275 catalog_manager.cc:4143] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf reported cstate change: config changed from index -1 to 110, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.356222 7390 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index -1 to 110, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.358561 8764 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s I1121 10:21:21.360325 7772 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index -1 to 110, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.362501 8020 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index -1 to 110, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.371117 7893 consensus_queue.cc:208] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.371680 7390 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch) I1121 10:21:21.371711 8022 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch) I1121 10:21:21.371888 8503 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s I1121 10:21:21.372027 8503 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s I1121 10:21:21.372797 7525 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch) I1121 10:21:21.373035 8926 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s W1121 10:21:21.373144 7832 consensus_peers.cc:473] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 605a58ce874c4170b0d24d922f017068. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:21.373183 7774 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch) I1121 10:21:21.373215 8266 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.373440 7518 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.373441 8336 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s I1121 10:21:21.373899 7774 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.374490 8022 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.375180 7275 catalog_manager.cc:4143] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf reported cstate change: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.375625 7390 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.383780 7893 consensus_queue.cc:208] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.384292 7527 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I1121 10:21:21.384348 8022 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) W1121 10:21:21.384373 7832 consensus_peers.cc:473] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c8cdc07bf6f4480b269ca34a9b3e426. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:21.384588 9006 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I1121 10:21:21.384598 7390 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I1121 10:21:21.384666 8504 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I1121 10:21:21.384907 9006 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I1121 10:21:21.385565 9006 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index -1 to 106, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.385915 8022 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.386555 7649 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch) I1121 10:21:21.387105 7275 catalog_manager.cc:4143] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index -1 to 106, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:21.387630 7527 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.387639 7390 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.392226 8504 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s I1121 10:21:21.393050 7649 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index -1 to 106, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.404381 7399 consensus_queue.cc:208] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.405154 7649 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch) I1121 10:21:21.405445 8970 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s I1121 10:21:21.406174 7893 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch) I1121 10:21:21.408579 7527 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch) W1121 10:21:21.409289 7334 consensus_peers.cc:473] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 -> Peer 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): Couldn't send request to peer 522e51b66e2c425091349ca2c9059244. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a9a3a26473340c381d502b8e9e1660c. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:21.409680 8022 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch) I1121 10:21:21.410897 8970 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s I1121 10:21:21.413173 8569 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s I1121 10:21:21.413841 8569 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s I1121 10:21:21.415977 8970 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.418339 7649 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.419911 8019 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.421383 7273 catalog_manager.cc:4143] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index -1 to 99, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.425846 7893 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.433426 7518 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index -1 to 99, NON_VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } } I1121 10:21:21.791963 9174 ts_tablet_manager.cc:721] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:21.792042 9174 tablet_copy_client.cc:243] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:43909 I1121 10:21:21.792217 7671 tablet_copy_service.cc:135] P f5bc6d2f9c574b4db9396f9f900c1141: Received BeginTabletCopySession request for tablet d84faff8cd0f433d8de8664fd0054d2d from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:43955) I1121 10:21:21.792244 7671 tablet_copy_service.cc:156] P f5bc6d2f9c574b4db9396f9f900c1141: Beginning new tablet copy session on tablet d84faff8cd0f433d8de8664fd0054d2d from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:43955: session id = 522e51b66e2c425091349ca2c9059244-d84faff8cd0f433d8de8664fd0054d2d I1121 10:21:21.793007 7671 tablet_copy_source_session.cc:204] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:21.793412 9174 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet d84faff8cd0f433d8de8664fd0054d2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:21.795243 9174 tablet_copy_client.cc:581] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:21.795387 9174 tablet_copy_client.cc:544] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:21.796605 9174 tablet_copy_client.cc:420] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:21.797344 9174 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:21.823391 9174 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=136 overwritten=0 applied=136 ignored=0} inserts{seen=833 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:21.823904 9174 tablet_bootstrap.cc:439] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:21.824039 9174 ts_tablet_manager.cc:1104] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.005s I1121 10:21:21.824236 9174 raft_consensus.cc:337] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.824262 9174 raft_consensus.cc:702] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:21.824329 9174 consensus_queue.cc:231] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 136, Last appended: 1.136, Last appended by leader: 136, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.825053 9174 ts_tablet_manager.cc:1132] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:21.825246 7671 tablet_copy_service.cc:325] P f5bc6d2f9c574b4db9396f9f900c1141: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-d84faff8cd0f433d8de8664fd0054d2d received from {username='slave'} at 127.5.70.7:43955 I1121 10:21:21.825273 7671 tablet_copy_service.cc:413] P f5bc6d2f9c574b4db9396f9f900c1141: ending tablet copy session 522e51b66e2c425091349ca2c9059244-d84faff8cd0f433d8de8664fd0054d2d on tablet d84faff8cd0f433d8de8664fd0054d2d with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:21.830018 9174 ts_tablet_manager.cc:721] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:21.830056 9174 tablet_copy_client.cc:243] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:43909 I1121 10:21:21.830261 7671 tablet_copy_service.cc:135] P f5bc6d2f9c574b4db9396f9f900c1141: Received BeginTabletCopySession request for tablet 787130b10eda4a71b2212bdb7ed2f140 from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:43955) I1121 10:21:21.830297 7671 tablet_copy_service.cc:156] P f5bc6d2f9c574b4db9396f9f900c1141: Beginning new tablet copy session on tablet 787130b10eda4a71b2212bdb7ed2f140 from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:43955: session id = 522e51b66e2c425091349ca2c9059244-787130b10eda4a71b2212bdb7ed2f140 I1121 10:21:21.830906 7671 tablet_copy_source_session.cc:204] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:21.831429 9174 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 787130b10eda4a71b2212bdb7ed2f140. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:21.834318 9174 tablet_copy_client.cc:581] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:21.834427 9174 tablet_copy_client.cc:544] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:21.836365 9174 tablet_copy_client.cc:420] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:21.837050 9174 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:21.854104 9174 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=104 overwritten=0 applied=104 ignored=0} inserts{seen=776 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:21.854367 9174 tablet_bootstrap.cc:439] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:21.854497 9174 ts_tablet_manager.cc:1104] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s I1121 10:21:21.854724 9174 raft_consensus.cc:337] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.854750 9174 raft_consensus.cc:702] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:21.854812 9174 consensus_queue.cc:231] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 104, Last appended: 1.104, Last appended by leader: 104, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.855132 9174 ts_tablet_manager.cc:1132] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:21.855314 7671 tablet_copy_service.cc:325] P f5bc6d2f9c574b4db9396f9f900c1141: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-787130b10eda4a71b2212bdb7ed2f140 received from {username='slave'} at 127.5.70.7:43955 I1121 10:21:21.855335 7671 tablet_copy_service.cc:413] P f5bc6d2f9c574b4db9396f9f900c1141: ending tablet copy session 522e51b66e2c425091349ca2c9059244-787130b10eda4a71b2212bdb7ed2f140 on tablet 787130b10eda4a71b2212bdb7ed2f140 with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:21.901188 9174 ts_tablet_manager.cc:721] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995) I1121 10:21:21.901250 9174 tablet_copy_client.cc:243] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:36995 I1121 10:21:21.901715 7422 tablet_copy_service.cc:135] P 63b9dc8cf49b47ebb644515af2f427e7: Received BeginTabletCopySession request for tablet 5a9a3a26473340c381d502b8e9e1660c from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:36379) I1121 10:21:21.901753 7422 tablet_copy_service.cc:156] P 63b9dc8cf49b47ebb644515af2f427e7: Beginning new tablet copy session on tablet 5a9a3a26473340c381d502b8e9e1660c from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:36379: session id = 522e51b66e2c425091349ca2c9059244-5a9a3a26473340c381d502b8e9e1660c I1121 10:21:21.902371 7422 tablet_copy_source_session.cc:204] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:21.902848 9174 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5a9a3a26473340c381d502b8e9e1660c. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:21.904386 9174 tablet_copy_client.cc:581] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:21.904503 9174 tablet_copy_client.cc:544] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:21.906764 9174 tablet_copy_client.cc:420] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:21.907574 9174 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:21.940939 9174 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=132 overwritten=0 applied=132 ignored=0} inserts{seen=751 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:21.941175 9174 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:21.941303 9174 ts_tablet_manager.cc:1104] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.009s I1121 10:21:21.941509 9174 raft_consensus.cc:337] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.941542 9174 raft_consensus.cc:702] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:21.941610 9174 consensus_queue.cc:231] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 132, Last appended: 1.132, Last appended by leader: 132, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:21.942065 9188 ts_tablet_manager.cc:721] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:21.942123 9188 tablet_copy_client.cc:243] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.5:36445 I1121 10:21:21.942483 7919 tablet_copy_service.cc:135] P 81d8bacc3ca64d628a255ad416a77c3d: Received BeginTabletCopySession request for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:53339) I1121 10:21:21.942512 7919 tablet_copy_service.cc:156] P 81d8bacc3ca64d628a255ad416a77c3d: Beginning new tablet copy session on tablet 5c8cdc07bf6f4480b269ca34a9b3e426 from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:53339: session id = 522e51b66e2c425091349ca2c9059244-5c8cdc07bf6f4480b269ca34a9b3e426 I1121 10:21:21.942564 9174 ts_tablet_manager.cc:1132] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:21.943037 7919 tablet_copy_source_session.cc:204] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:21.943172 7422 tablet_copy_service.cc:325] P 63b9dc8cf49b47ebb644515af2f427e7: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-5a9a3a26473340c381d502b8e9e1660c received from {username='slave'} at 127.5.70.7:36379 I1121 10:21:21.943199 7422 tablet_copy_service.cc:413] P 63b9dc8cf49b47ebb644515af2f427e7: ending tablet copy session 522e51b66e2c425091349ca2c9059244-5a9a3a26473340c381d502b8e9e1660c on tablet 5a9a3a26473340c381d502b8e9e1660c with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:21.943912 9188 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5c8cdc07bf6f4480b269ca34a9b3e426. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:21.945847 9188 tablet_copy_client.cc:581] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:21.945977 9188 tablet_copy_client.cc:544] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:21.947244 9188 tablet_copy_client.cc:420] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:21.947929 9188 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:21.981460 9174 ts_tablet_manager.cc:721] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: Initiating tablet copy from peer 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:21.983265 9174 tablet_copy_client.cc:243] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.5:36445 I1121 10:21:21.986413 7919 tablet_copy_service.cc:135] P 81d8bacc3ca64d628a255ad416a77c3d: Received BeginTabletCopySession request for tablet 605a58ce874c4170b0d24d922f017068 from peer 522e51b66e2c425091349ca2c9059244 ({username='slave'} at 127.5.70.7:53339) I1121 10:21:21.986457 7919 tablet_copy_service.cc:156] P 81d8bacc3ca64d628a255ad416a77c3d: Beginning new tablet copy session on tablet 605a58ce874c4170b0d24d922f017068 from peer 522e51b66e2c425091349ca2c9059244 at {username='slave'} at 127.5.70.7:53339: session id = 522e51b66e2c425091349ca2c9059244-605a58ce874c4170b0d24d922f017068 I1121 10:21:21.987046 7919 tablet_copy_source_session.cc:204] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:21.987845 9174 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 605a58ce874c4170b0d24d922f017068. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:21.989389 9174 tablet_copy_client.cc:581] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 0 data blocks... I1121 10:21:21.989503 9174 tablet_copy_client.cc:544] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: tablet copy: Starting download of 1 WAL segments... I1121 10:21:21.990635 9174 tablet_copy_client.cc:420] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:21.992589 9174 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: Bootstrap starting. I1121 10:21:22.016381 9174 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=121 overwritten=0 applied=121 ignored=0} inserts{seen=793 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:22.016772 9174 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:22.016947 9174 ts_tablet_manager.cc:1104] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.006s I1121 10:21:22.017172 9174 raft_consensus.cc:337] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:22.017261 9174 raft_consensus.cc:702] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:22.017374 9174 consensus_queue.cc:231] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 121, Last appended: 1.121, Last appended by leader: 121, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:22.018359 9174 ts_tablet_manager.cc:1132] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:22.018760 7919 tablet_copy_service.cc:325] P 81d8bacc3ca64d628a255ad416a77c3d: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-605a58ce874c4170b0d24d922f017068 received from {username='slave'} at 127.5.70.7:53339 I1121 10:21:22.018785 7919 tablet_copy_service.cc:413] P 81d8bacc3ca64d628a255ad416a77c3d: ending tablet copy session 522e51b66e2c425091349ca2c9059244-605a58ce874c4170b0d24d922f017068 on tablet 605a58ce874c4170b0d24d922f017068 with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:22.053401 9188 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: Bootstrap replayed 1/1 log segments. Stats: ops{read=136 overwritten=0 applied=136 ignored=0} inserts{seen=755 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:22.053871 9188 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: Bootstrap complete. I1121 10:21:22.054014 9188 ts_tablet_manager.cc:1104] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.007s I1121 10:21:22.054231 9188 raft_consensus.cc:337] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:22.054258 9188 raft_consensus.cc:702] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 522e51b66e2c425091349ca2c9059244, State: Initialized, Role: LEARNER I1121 10:21:22.054322 9188 consensus_queue.cc:231] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 136, Last appended: 1.136, Last appended by leader: 136, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: true } } I1121 10:21:22.054888 9188 ts_tablet_manager.cc:1132] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:22.055333 7919 tablet_copy_service.cc:325] P 81d8bacc3ca64d628a255ad416a77c3d: Request end of tablet copy session 522e51b66e2c425091349ca2c9059244-5c8cdc07bf6f4480b269ca34a9b3e426 received from {username='slave'} at 127.5.70.7:53339 I1121 10:21:22.055364 7919 tablet_copy_service.cc:413] P 81d8bacc3ca64d628a255ad416a77c3d: ending tablet copy session 522e51b66e2c425091349ca2c9059244-5c8cdc07bf6f4480b269ca34a9b3e426 on tablet 5c8cdc07bf6f4480b269ca34a9b3e426 with peer 522e51b66e2c425091349ca2c9059244 I1121 10:21:22.243643 8660 raft_consensus.cc:1123] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.109->[1.110-1.162] Dedup: 1.136->[1.137-1.162] I1121 10:21:22.248872 8273 raft_consensus.cc:971] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:22.249081 8273 consensus_queue.cc:208] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 163, Committed index: 163, Last appended: 1.163, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.250227 7771 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 163. Preceding OpId from leader: term: 1 index: 164. (index mismatch) I1121 10:21:22.250473 8661 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 163. Preceding OpId from leader: term: 1 index: 164. (index mismatch) I1121 10:21:22.250735 8920 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 163, Time since last communication: 0.000s I1121 10:21:22.251106 8920 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 163, Time since last communication: 0.000s I1121 10:21:22.251111 7527 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 163. Preceding OpId from leader: term: 1 index: 164. (index mismatch) I1121 10:21:22.252115 7399 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 163. Preceding OpId from leader: term: 1 index: 164. (index mismatch) I1121 10:21:22.253361 8920 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 163, Time since last communication: 0.000s I1121 10:21:22.254472 8762 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 163, Time since last communication: 0.000s I1121 10:21:22.255969 8020 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 163. Preceding OpId from leader: term: 1 index: 164. (index mismatch) I1121 10:21:22.258455 8762 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 164, Last known committed idx: 163, Time since last communication: 0.000s I1121 10:21:22.259513 8764 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.164: config changed from index 110 to 164, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.261368 7399 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.164: config changed from index 110 to 164, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.263490 7273 catalog_manager.cc:4143] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 110 to 164, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.266358 8019 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.164: config changed from index 110 to 164, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.271376 7773 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.164: config changed from index 110 to 164, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.271759 8659 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.164: config changed from index 110 to 164, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.272830 7525 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.164: config changed from index 110 to 164, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 164 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.273773 8661 raft_consensus.cc:1123] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.82->[1.83-1.123] Dedup: 1.104->[1.105-1.123] I1121 10:21:22.278645 7649 consensus_queue.cc:208] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 165, Committed index: 165, Last appended: 1.165, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.279523 7771 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch) I1121 10:21:22.282637 8019 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch) I1121 10:21:22.283040 8661 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch) I1121 10:21:22.284157 8762 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s I1121 10:21:22.285089 8762 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s I1121 10:21:22.285173 7527 raft_consensus.cc:1181] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch) I1121 10:21:22.286577 8764 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s I1121 10:21:22.287266 8762 consensus_queue.cc:984] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s I1121 10:21:22.289096 9172 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.166: config changed from index 164 to 166, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.290343 7261 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet d84faff8cd0f433d8de8664fd0054d2d with cas_config_opid_index 164: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:22.290436 8020 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 164 to 166, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.291455 8762 raft_consensus.cc:971] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:22.291636 8762 consensus_queue.cc:208] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.124, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.292294 7399 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch) I1121 10:21:22.292379 7772 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 164 to 166, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.292651 7275 catalog_manager.cc:4143] T d84faff8cd0f433d8de8664fd0054d2d P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index 164 to 166, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.292670 9172 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s I1121 10:21:22.293402 8661 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch) I1121 10:21:22.293648 7896 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch) I1121 10:21:22.293867 7772 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch) I1121 10:21:22.294075 7370 tablet_service.cc:1067] Processing DeleteTablet for tablet d84faff8cd0f433d8de8664fd0054d2d with delete_type TABLET_DATA_TOMBSTONED (TS 63b9dc8cf49b47ebb644515af2f427e7 not found in new config with opid_index 166) from {username='slave'} at 127.0.0.1:34708 I1121 10:21:22.294221 9250 tablet_replica.cc:282] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: stopping tablet replica I1121 10:21:22.294271 9250 raft_consensus.cc:2156] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:22.294292 9250 raft_consensus.cc:2183] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:22.294508 7525 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 164 to 166, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.295363 9250 ts_tablet_manager.cc:1457] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:22.295403 8273 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s I1121 10:21:22.295796 9172 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s I1121 10:21:22.296079 8661 raft_consensus.cc:2801] T d84faff8cd0f433d8de8664fd0054d2d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 164 to 166, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.296213 8400 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s I1121 10:21:22.297019 7525 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch) I1121 10:21:22.297480 8764 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s I1121 10:21:22.299609 9250 ts_tablet_manager.cc:1470] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.165 I1121 10:21:22.299649 9250 log.cc:1153] T d84faff8cd0f433d8de8664fd0054d2d P 63b9dc8cf49b47ebb644515af2f427e7: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/d84faff8cd0f433d8de8664fd0054d2d I1121 10:21:22.300192 7262 catalog_manager.cc:3512] TS 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995): tablet d84faff8cd0f433d8de8664fd0054d2d (table rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]) successfully deleted I1121 10:21:22.303242 8273 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.125: config changed from index 83 to 125, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.304067 7771 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 83 to 125, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.304672 8661 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 83 to 125, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.304670 7399 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 83 to 125, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.305896 7278 catalog_manager.cc:4143] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index 83 to 125, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:22.305951 7899 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 83 to 125, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.318135 7525 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 83 to 125, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.318424 7651 consensus_queue.cc:208] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 1.125, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.318838 7399 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch) I1121 10:21:22.319061 8400 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s I1121 10:21:22.321306 8659 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch) I1121 10:21:22.322525 7527 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch) I1121 10:21:22.322722 7774 raft_consensus.cc:1181] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 126. (index mismatch) I1121 10:21:22.323385 8400 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s I1121 10:21:22.324563 8400 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s I1121 10:21:22.325453 8400 consensus_queue.cc:984] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s I1121 10:21:22.328919 8659 raft_consensus.cc:1123] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.98->[1.99-1.165] Dedup: 1.132->[1.133-1.165] I1121 10:21:22.329084 8291 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.126: config changed from index 125 to 126, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.330143 7261 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 787130b10eda4a71b2212bdb7ed2f140 with cas_config_opid_index 125: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:22.330893 7399 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 125 to 126, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.332844 8659 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 125 to 126, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.333070 7773 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 125 to 126, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.333106 7527 raft_consensus.cc:2801] T 787130b10eda4a71b2212bdb7ed2f140 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.126: config changed from index 125 to 126, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.334300 7275 catalog_manager.cc:4143] T 787130b10eda4a71b2212bdb7ed2f140 P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 125 to 126, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 126 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.337761 7874 tablet_service.cc:1067] Processing DeleteTablet for tablet 787130b10eda4a71b2212bdb7ed2f140 with delete_type TABLET_DATA_TOMBSTONED (TS 81d8bacc3ca64d628a255ad416a77c3d not found in new config with opid_index 126) from {username='slave'} at 127.0.0.1:37730 I1121 10:21:22.343158 8570 raft_consensus.cc:971] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:22.343432 8570 consensus_queue.cc:208] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 165, Committed index: 165, Last appended: 1.165, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.343943 8020 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch) I1121 10:21:22.344120 8568 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s I1121 10:21:22.344393 8661 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch) I1121 10:21:22.344774 9253 tablet_replica.cc:282] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d: stopping tablet replica I1121 10:21:22.344904 9253 raft_consensus.cc:2156] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:22.344936 7899 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch) I1121 10:21:22.345093 7649 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch) I1121 10:21:22.345098 9253 raft_consensus.cc:2183] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:22.345388 8568 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s I1121 10:21:22.345669 9253 ts_tablet_manager.cc:1457] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:22.347856 9253 ts_tablet_manager.cc:1470] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.125 I1121 10:21:22.347941 9253 log.cc:1153] T 787130b10eda4a71b2212bdb7ed2f140 P 81d8bacc3ca64d628a255ad416a77c3d: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal/wals/787130b10eda4a71b2212bdb7ed2f140 I1121 10:21:22.349699 7263 catalog_manager.cc:3512] TS 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445): tablet 787130b10eda4a71b2212bdb7ed2f140 (table rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]) successfully deleted I1121 10:21:22.354948 7525 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 165. Preceding OpId from leader: term: 1 index: 166. (index mismatch) I1121 10:21:22.356242 8970 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s I1121 10:21:22.356426 8562 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s I1121 10:21:22.356602 8560 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 166, Last known committed idx: 165, Time since last communication: 0.000s I1121 10:21:22.362139 8569 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.166: config changed from index 99 to 166, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.363513 8658 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 99 to 166, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.364071 7651 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 99 to 166, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.364980 7274 catalog_manager.cc:4143] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 99 to 166, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.366904 7899 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 99 to 166, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.368319 7525 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 99 to 166, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.369505 8023 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.166: config changed from index 99 to 166, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 166 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.371043 7399 consensus_queue.cc:208] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 166, Committed index: 166, Last appended: 1.166, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.371632 8658 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch) I1121 10:21:22.371757 7899 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch) I1121 10:21:22.372570 8023 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch) I1121 10:21:22.373771 8569 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s I1121 10:21:22.373790 7651 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 167. (index mismatch) I1121 10:21:22.375048 8569 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s I1121 10:21:22.376222 8970 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.167: config changed from index 166 to 167, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.377035 7651 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 166 to 167, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.377725 7262 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 5a9a3a26473340c381d502b8e9e1660c with cas_config_opid_index 166: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:22.378185 8560 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s I1121 10:21:22.378273 8560 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 167, Last known committed idx: 166, Time since last communication: 0.000s I1121 10:21:22.378554 7274 catalog_manager.cc:4143] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 166 to 167, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:22.378749 8659 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 166 to 167, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.380352 7501 tablet_service.cc:1067] Processing DeleteTablet for tablet 5a9a3a26473340c381d502b8e9e1660c with delete_type TABLET_DATA_TOMBSTONED (TS 41118f5c2e8f48179ffbebbfb6da2edf not found in new config with opid_index 167) from {username='slave'} at 127.0.0.1:49436 I1121 10:21:22.381503 9254 tablet_replica.cc:282] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf: stopping tablet replica I1121 10:21:22.381567 9254 raft_consensus.cc:2156] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:22.381592 9254 raft_consensus.cc:2183] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:22.382220 8023 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 166 to 167, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.385879 7896 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.167: config changed from index 166 to 167, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 167 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.387262 9254 ts_tablet_manager.cc:1457] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:22.403411 9254 ts_tablet_manager.cc:1470] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.166 I1121 10:21:22.403468 9254 log.cc:1153] T 5a9a3a26473340c381d502b8e9e1660c P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/5a9a3a26473340c381d502b8e9e1660c I1121 10:21:22.406642 7263 catalog_manager.cc:3512] TS 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291): tablet 5a9a3a26473340c381d502b8e9e1660c (table rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]) successfully deleted I1121 10:21:22.430392 8661 raft_consensus.cc:1123] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.105->[1.106-1.170] Dedup: 1.136->[1.137-1.170] I1121 10:21:22.444396 9215 raft_consensus.cc:971] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:22.444672 9215 consensus_queue.cc:208] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 172, Committed index: 172, Last appended: 1.172, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.446622 8019 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 172. Preceding OpId from leader: term: 1 index: 173. (index mismatch) I1121 10:21:22.446898 8503 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 173, Last known committed idx: 172, Time since last communication: 0.000s I1121 10:21:22.447830 7649 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 172. Preceding OpId from leader: term: 1 index: 173. (index mismatch) I1121 10:21:22.448472 8503 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 173, Last known committed idx: 172, Time since last communication: 0.000s I1121 10:21:22.449570 8658 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 171. Preceding OpId from leader: term: 1 index: 173. (index mismatch) I1121 10:21:22.449656 7518 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 172. Preceding OpId from leader: term: 1 index: 173. (index mismatch) I1121 10:21:22.449802 7399 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 172. Preceding OpId from leader: term: 1 index: 173. (index mismatch) I1121 10:21:22.450486 8926 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 173, Last known committed idx: 172, Time since last communication: 0.000s I1121 10:21:22.451284 8926 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 173, Last known committed idx: 172, Time since last communication: 0.000s I1121 10:21:22.451356 8266 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 173, Last known committed idx: 171, Time since last communication: 0.000s I1121 10:21:22.454608 8658 raft_consensus.cc:1123] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.171->[1.172-1.174] Dedup: 1.172->[1.173-1.174] I1121 10:21:22.454814 8266 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.173: config changed from index 106 to 173, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.456317 8020 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.173: config changed from index 106 to 173, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.456933 7525 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.173: config changed from index 106 to 173, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.457868 7274 catalog_manager.cc:4143] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index 106 to 173, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.458128 7649 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.173: config changed from index 106 to 173, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.460009 7402 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.173: config changed from index 106 to 173, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.462386 7896 consensus_queue.cc:208] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 174, Committed index: 174, Last appended: 1.175, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.462594 8661 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.173: config changed from index 106 to 173, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 173 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.463346 7649 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 176. (index mismatch) I1121 10:21:22.463769 8658 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 174. Preceding OpId from leader: term: 1 index: 176. (index mismatch) I1121 10:21:22.465349 8020 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 176. (index mismatch) I1121 10:21:22.465562 8266 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s I1121 10:21:22.465785 8266 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s I1121 10:21:22.466295 8926 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s I1121 10:21:22.466490 7402 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 175. Preceding OpId from leader: term: 1 index: 176. (index mismatch) I1121 10:21:22.467403 8266 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 176, Last known committed idx: 174, Time since last communication: 0.000s I1121 10:21:22.468338 8926 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.176: config changed from index 173 to 176, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.472162 8019 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 173 to 176, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.473425 7649 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 173 to 176, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.473644 7274 catalog_manager.cc:4143] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index 173 to 176, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.474581 7501 tablet_service.cc:1067] Processing DeleteTablet for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 with delete_type TABLET_DATA_TOMBSTONED (TS 41118f5c2e8f48179ffbebbfb6da2edf not found in new config with opid_index 176) from {username='slave'} at 127.0.0.1:49436 I1121 10:21:22.474838 7402 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 173 to 176, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.474997 9254 tablet_replica.cc:282] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: stopping tablet replica I1121 10:21:22.475111 9254 raft_consensus.cc:2156] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:22.475143 9254 pending_rounds.cc:64] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: Trying to abort 1 pending transactions. I1121 10:21:22.475257 9254 pending_rounds.cc:70] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: Aborting transaction as it isn't in flight: id { term: 1 index: 175 } timestamp: 6448462571348959232 op_type: WRITE_OP write_request { tablet_id: "5c8cdc07bf6f4480b269ca34a9b3e426" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\257\300\356:\"\205u\234\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "ac7b2d9d95dfac1e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6448462571319336960 authz_token { token_data: "\010\313\314\331\356\005\"3\n\005slave\022*\n 692cf724e38648ed81e17afafb0bbf35\020\001\030\001 \001(\001" signature: "\267\340(\312j\024+\313{rGp\320\271\346\201,!\203+ :\363N\223`D=Q\227\'\221\337H\310\2524\215\031.G\343N\206\217\261P\377`\357\t\006\002\204+Pz\333\245\025t\025Xn" signing_key_seq_num: 0 } } request_id { client_id: "93b2303c34784b7d88db2a408f059a79" seq_no: 439 first_incomplete_seq_no: 439 attempt_no: 0 } I1121 10:21:22.475342 9254 raft_consensus.cc:2183] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:22.475617 8659 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.176: config changed from index 173 to 176, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 176 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.475697 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 with cas_config_opid_index 173: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:22.475773 9254 ts_tablet_manager.cc:1457] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:22.477519 8658 raft_consensus.cc:1123] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.82->[1.83-1.143] Dedup: 1.121->[1.122-1.143] I1121 10:21:22.482241 9254 ts_tablet_manager.cc:1470] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.175 I1121 10:21:22.482280 9254 log.cc:1153] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/5c8cdc07bf6f4480b269ca34a9b3e426 I1121 10:21:22.483862 7263 catalog_manager.cc:3512] TS 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291): tablet 5c8cdc07bf6f4480b269ca34a9b3e426 (table rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]) successfully deleted I1121 10:21:22.488992 8266 raft_consensus.cc:971] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d: attempting to promote NON_VOTER 522e51b66e2c425091349ca2c9059244 to VOTER I1121 10:21:22.489205 8266 consensus_queue.cc:208] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 143, Committed index: 143, Last appended: 1.144, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.489725 8661 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 LEARNER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 144. Preceding OpId from leader: term: 1 index: 145. (index mismatch) I1121 10:21:22.490137 8327 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 143, Time since last communication: 0.000s I1121 10:21:22.490190 7518 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 144. Preceding OpId from leader: term: 1 index: 145. (index mismatch) I1121 10:21:22.490388 8327 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 143, Time since last communication: 0.000s I1121 10:21:22.491230 8019 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 144. Preceding OpId from leader: term: 1 index: 145. (index mismatch) I1121 10:21:22.491437 8327 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 143, Time since last communication: 0.000s I1121 10:21:22.492213 7771 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 144. Preceding OpId from leader: term: 1 index: 145. (index mismatch) I1121 10:21:22.492213 7402 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 144. Preceding OpId from leader: term: 1 index: 145. (index mismatch) I1121 10:21:22.493032 8926 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 143, Time since last communication: 0.000s I1121 10:21:22.493084 8327 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 145, Last known committed idx: 143, Time since last communication: 0.000s I1121 10:21:22.494140 9006 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.145: config changed from index 83 to 145, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.494594 8661 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.145: config changed from index 83 to 145, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.495463 8020 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.145: config changed from index 83 to 145, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.495913 7518 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.145: config changed from index 83 to 145, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.495963 7273 catalog_manager.cc:4143] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 83 to 145, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.500875 7399 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.145: config changed from index 83 to 145, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.502015 7771 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.145: config changed from index 83 to 145, 522e51b66e2c425091349ca2c9059244 (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 145 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.502164 7893 consensus_queue.cc:208] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 145, Committed index: 145, Last appended: 1.145, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } I1121 10:21:22.502967 8019 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch) I1121 10:21:22.503195 8327 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s I1121 10:21:22.504154 8661 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch) I1121 10:21:22.504257 7399 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch) I1121 10:21:22.504649 8327 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s I1121 10:21:22.505090 8327 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s I1121 10:21:22.505892 8926 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.146: config changed from index 145 to 146, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.506381 7774 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch) I1121 10:21:22.506898 9215 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s I1121 10:21:22.507165 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 605a58ce874c4170b0d24d922f017068 with cas_config_opid_index 145: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:22.507593 8659 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 145 to 146, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.507604 7771 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 145 to 146, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.509528 7275 catalog_manager.cc:4143] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a reported cstate change: config changed from index 145 to 146, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.510952 7501 tablet_service.cc:1067] Processing DeleteTablet for tablet 605a58ce874c4170b0d24d922f017068 with delete_type TABLET_DATA_TOMBSTONED (TS 41118f5c2e8f48179ffbebbfb6da2edf not found in new config with opid_index 146) from {username='slave'} at 127.0.0.1:49436 I1121 10:21:22.511036 9254 tablet_replica.cc:282] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf: stopping tablet replica I1121 10:21:22.511092 9254 raft_consensus.cc:2156] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:22.511119 9254 raft_consensus.cc:2183] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:22.511885 9254 ts_tablet_manager.cc:1457] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:22.512179 8019 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 145 to 146, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.514569 9254 ts_tablet_manager.cc:1470] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.145 I1121 10:21:22.514582 7399 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 145 to 146, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } } I1121 10:21:22.514608 9254 log.cc:1153] T 605a58ce874c4170b0d24d922f017068 P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/605a58ce874c4170b0d24d922f017068 I1121 10:21:22.515028 7263 catalog_manager.cc:3512] TS 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291): tablet 605a58ce874c4170b0d24d922f017068 (table rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]) successfully deleted I1121 10:21:22.704771 7899 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 170, Committed index: 170, Last appended: 1.170, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } I1121 10:21:22.705245 8661 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch) I1121 10:21:22.705579 7775 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch) I1121 10:21:22.706174 8503 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s I1121 10:21:22.706562 8021 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch) I1121 10:21:22.706559 8503 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s I1121 10:21:22.707221 8503 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s I1121 10:21:22.708645 8926 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.171: config changed from index 87 to 171, NON_VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) added. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } } I1121 10:21:22.711618 8657 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 87 to 171, NON_VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) added. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } } W1121 10:21:22.712378 7831 consensus_peers.cc:473] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995): Couldn't send request to peer 63b9dc8cf49b47ebb644515af2f427e7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cd3c7862a7c420cb0b87bd87a770d84. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:22.712894 8019 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 87 to 171, NON_VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) added. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } } I1121 10:21:22.713062 7275 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 87 to 171, NON_VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) added. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } } I1121 10:21:22.714056 7775 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 87 to 171, NON_VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) added. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } } I1121 10:21:22.714953 7649 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch) I1121 10:21:22.718019 8926 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 169, Time since last communication: 0.000s I1121 10:21:22.721698 7649 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 87 to 171, NON_VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) added. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } } I1121 10:21:23.299945 9348 ts_tablet_manager.cc:721] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: Initiating tablet copy from peer 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:23.300010 9348 tablet_copy_client.cc:243] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.5:36445 I1121 10:21:23.300282 7919 tablet_copy_service.cc:135] P 81d8bacc3ca64d628a255ad416a77c3d: Received BeginTabletCopySession request for tablet 7cd3c7862a7c420cb0b87bd87a770d84 from peer 63b9dc8cf49b47ebb644515af2f427e7 ({username='slave'} at 127.5.70.1:58573) I1121 10:21:23.300312 7919 tablet_copy_service.cc:156] P 81d8bacc3ca64d628a255ad416a77c3d: Beginning new tablet copy session on tablet 7cd3c7862a7c420cb0b87bd87a770d84 from peer 63b9dc8cf49b47ebb644515af2f427e7 at {username='slave'} at 127.5.70.1:58573: session id = 63b9dc8cf49b47ebb644515af2f427e7-7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:23.301200 7919 tablet_copy_source_session.cc:204] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:23.304342 9348 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7cd3c7862a7c420cb0b87bd87a770d84. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:23.321311 9348 tablet_copy_client.cc:581] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: tablet copy: Starting download of 0 data blocks... I1121 10:21:23.321487 9348 tablet_copy_client.cc:544] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: tablet copy: Starting download of 1 WAL segments... I1121 10:21:23.332329 9348 tablet_copy_client.cc:420] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:23.334802 9348 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap starting. I1121 10:21:23.371701 9348 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=202 overwritten=0 applied=202 ignored=0} inserts{seen=896 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:23.372179 9348 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: Bootstrap complete. I1121 10:21:23.372349 9348 ts_tablet_manager.cc:1104] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.007s I1121 10:21:23.372743 9348 raft_consensus.cc:337] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } I1121 10:21:23.372776 9348 raft_consensus.cc:702] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 63b9dc8cf49b47ebb644515af2f427e7, State: Initialized, Role: LEARNER I1121 10:21:23.372843 9348 consensus_queue.cc:231] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 202, Last appended: 1.202, Last appended by leader: 202, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } I1121 10:21:23.374873 9348 ts_tablet_manager.cc:1132] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I1121 10:21:23.378659 7919 tablet_copy_service.cc:325] P 81d8bacc3ca64d628a255ad416a77c3d: Request end of tablet copy session 63b9dc8cf49b47ebb644515af2f427e7-7cd3c7862a7c420cb0b87bd87a770d84 received from {username='slave'} at 127.5.70.1:58573 I1121 10:21:23.378685 7919 tablet_copy_service.cc:413] P 81d8bacc3ca64d628a255ad416a77c3d: ending tablet copy session 63b9dc8cf49b47ebb644515af2f427e7-7cd3c7862a7c420cb0b87bd87a770d84 on tablet 7cd3c7862a7c420cb0b87bd87a770d84 with peer 63b9dc8cf49b47ebb644515af2f427e7 I1121 10:21:23.795521 7399 raft_consensus.cc:1123] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.170->[1.171-1.232] Dedup: 1.202->[1.203-1.232] I1121 10:21:23.806241 8926 raft_consensus.cc:971] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: attempting to promote NON_VOTER 63b9dc8cf49b47ebb644515af2f427e7 to VOTER I1121 10:21:23.806468 8926 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 1.233, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } I1121 10:21:23.807305 8022 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch) I1121 10:21:23.808466 7390 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEARNER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch) I1121 10:21:23.808809 7772 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch) I1121 10:21:23.809134 8503 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s I1121 10:21:23.809224 8266 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s I1121 10:21:23.809715 7649 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch) I1121 10:21:23.809911 8503 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s I1121 10:21:23.810351 8658 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 233. Preceding OpId from leader: term: 1 index: 234. (index mismatch) I1121 10:21:23.811712 8503 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s I1121 10:21:23.813719 8327 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s I1121 10:21:23.816301 8336 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.234: config changed from index 171 to 234, 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) changed from NON_VOTER to VOTER. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } I1121 10:21:23.816380 8022 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 171 to 234, 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) changed from NON_VOTER to VOTER. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } I1121 10:21:23.818161 7278 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index 171 to 234, 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:23.818568 8659 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 171 to 234, 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) changed from NON_VOTER to VOTER. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } I1121 10:21:23.818790 7390 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 171 to 234, 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) changed from NON_VOTER to VOTER. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } I1121 10:21:23.819658 7896 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 235, Committed index: 235, Last appended: 1.235, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } I1121 10:21:23.821516 7771 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.234: config changed from index 171 to 234, 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) changed from NON_VOTER to VOTER. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } I1121 10:21:23.821728 8658 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch) I1121 10:21:23.822155 8503 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s I1121 10:21:23.822847 7775 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch) I1121 10:21:23.823601 8022 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch) I1121 10:21:23.824370 8327 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s I1121 10:21:23.824990 8266 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s I1121 10:21:23.826138 8327 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.236: config changed from index 234 to 236, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } I1121 10:21:23.826252 8022 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 234 to 236, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } I1121 10:21:23.827466 7275 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index 234 to 236, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:23.828613 7628 tablet_service.cc:1067] Processing DeleteTablet for tablet 7cd3c7862a7c420cb0b87bd87a770d84 with delete_type TABLET_DATA_TOMBSTONED (TS f5bc6d2f9c574b4db9396f9f900c1141 not found in new config with opid_index 236) from {username='slave'} at 127.0.0.1:39024 I1121 10:21:23.828713 8658 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 234 to 236, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } I1121 10:21:23.829077 7390 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 235. Preceding OpId from leader: term: 1 index: 236. (index mismatch) I1121 10:21:23.830253 9379 tablet_replica.cc:282] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: stopping tablet replica I1121 10:21:23.830307 9379 raft_consensus.cc:2156] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:23.830327 9379 pending_rounds.cc:64] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: Trying to abort 2 pending transactions. I1121 10:21:23.830438 9379 pending_rounds.cc:70] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: Aborting transaction as it isn't in flight: id { term: 1 index: 235 } timestamp: 6448462576880410624 op_type: WRITE_OP write_request { tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000q\001\025Q\003J:\346\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "c6cc5e4d2686296f" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6448462576856109056 authz_token { token_data: "\010\313\314\331\356\005\"3\n\005slave\022*\n a6818c1427a0476183a4cee5788608f5\020\001\030\001 \001(\001" signature: "\263\342U\262`m\357u>#\255b\3762\274\337$\201\375\324\243\231\340\376\357\361\347}\\\226\353R\\\260z\337_\211\275\345\263{\177\343\221\226+\2577O3\352\r\210\260\362x\200\314S:\177h\202" signing_key_seq_num: 0 } } request_id { client_id: "1e061c6c3cb44fb5ae3c235bfc8b3486" seq_no: 794 first_incomplete_seq_no: 794 attempt_no: 0 } I1121 10:21:23.830612 9379 pending_rounds.cc:70] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: Aborting transaction as it isn't in flight: id { term: 1 index: 234 } timestamp: 6448462576870690816 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7cd3c7862a7c420cb0b87bd87a770d84" old_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: NON_VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: true } } } new_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } } I1121 10:21:23.830646 9379 raft_consensus.cc:2763] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Aborting config change with OpId 1.234: Aborted: Transaction aborted I1121 10:21:23.830662 9379 raft_consensus.cc:2736] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Transaction aborted I1121 10:21:23.830682 9379 raft_consensus.cc:2183] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:23.830796 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 7cd3c7862a7c420cb0b87bd87a770d84 with cas_config_opid_index 234: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:23.831478 9379 ts_tablet_manager.cc:1457] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:23.832113 8336 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 236, Last known committed idx: 235, Time since last communication: 0.000s I1121 10:21:23.833745 7390 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 234 to 236, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } I1121 10:21:23.836005 9379 ts_tablet_manager.cc:1470] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.235 I1121 10:21:23.836201 9379 log.cc:1153] T 7cd3c7862a7c420cb0b87bd87a770d84 P f5bc6d2f9c574b4db9396f9f900c1141: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:23.837229 7261 catalog_manager.cc:3512] TS f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909): tablet 7cd3c7862a7c420cb0b87bd87a770d84 (table rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]) successfully deleted I1121 10:21:23.847396 7775 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.236: config changed from index 234 to 236, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 236 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } } Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 913850c84b1540c1945d797659799e2b | 127.5.70.62:38839 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 1b3038ad6c564f438aa18383cddc769a | 127.5.70.4:32993 | HEALTHY | 41118f5c2e8f48179ffbebbfb6da2edf | 127.5.70.2:39291 | HEALTHY | 522e51b66e2c425091349ca2c9059244 | 127.5.70.7:38147 | HEALTHY | 63b9dc8cf49b47ebb644515af2f427e7 | 127.5.70.1:36995 | HEALTHY | 81d8bacc3ca64d628a255ad416a77c3d | 127.5.70.5:36445 | HEALTHY | c9e9c35ff1cb4b68ba7cf5c037a98347 | 127.5.70.6:32833 | HEALTHY | f5bc6d2f9c574b4db9396f9f900c1141 | 127.5.70.3:43909 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 7 Flag | Value | Tags | Tablet Server ------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 7 server(s) checked local_ip_for_outbound_sockets | 127.5.70.1 | experimental | 127.5.70.1:36995 local_ip_for_outbound_sockets | 127.5.70.2 | experimental | 127.5.70.2:39291 local_ip_for_outbound_sockets | 127.5.70.3 | experimental | 127.5.70.3:43909 local_ip_for_outbound_sockets | 127.5.70.4 | experimental | 127.5.70.4:32993 local_ip_for_outbound_sockets | 127.5.70.5 | experimental | 127.5.70.5:36445 local_ip_for_outbound_sockets | 127.5.70.6 | experimental | 127.5.70.6:32833 local_ip_for_outbound_sockets | 127.5.70.7 | experimental | 127.5.70.7:38147 never_fsync | true | unsafe,advanced | all 7 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked server_dump_info_format | pb | hidden | all 7 server(s) checked server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.70.1:36995 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.70.2:39291 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.70.3:43909 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.70.4:32993 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.70.5:36445 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.70.6:32833 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.70.7:38147 Version Summary Version | Servers -----------------+------------------------- 1.12.0-SNAPSHOT | all 8 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ------------------------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0 rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 11 First Quartile | 11 Median | 11 Third Quartile | 12 Maximum | 12 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 7 Tables | 6 Tablets | 16 Replicas | 80 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1121 10:21:24.918972 5400 log_verifier.cc:121] Checking tablet 133fea61ffe5433785a3fe3736a2a668 I1121 10:21:24.992188 5400 log_verifier.cc:172] Verified matching terms for 273 ops in tablet 133fea61ffe5433785a3fe3736a2a668 I1121 10:21:24.992339 5400 log_verifier.cc:121] Checking tablet 169c4cf587a949748d1734c65258b7a7 I1121 10:21:25.073554 5400 log_verifier.cc:172] Verified matching terms for 341 ops in tablet 169c4cf587a949748d1734c65258b7a7 I1121 10:21:25.073612 5400 log_verifier.cc:121] Checking tablet 55d18b1e7ede4cb28f5dfc4ded166a30 I1121 10:21:25.150811 5400 log_verifier.cc:172] Verified matching terms for 252 ops in tablet 55d18b1e7ede4cb28f5dfc4ded166a30 I1121 10:21:25.150878 5400 log_verifier.cc:121] Checking tablet 5a9a3a26473340c381d502b8e9e1660c I1121 10:21:25.229404 5400 log_verifier.cc:172] Verified matching terms for 300 ops in tablet 5a9a3a26473340c381d502b8e9e1660c I1121 10:21:25.229455 5400 log_verifier.cc:121] Checking tablet 5c8cdc07bf6f4480b269ca34a9b3e426 I1121 10:21:25.307611 5400 log_verifier.cc:172] Verified matching terms for 270 ops in tablet 5c8cdc07bf6f4480b269ca34a9b3e426 I1121 10:21:25.307662 5400 log_verifier.cc:121] Checking tablet 605a58ce874c4170b0d24d922f017068 I1121 10:21:25.389914 5400 log_verifier.cc:172] Verified matching terms for 281 ops in tablet 605a58ce874c4170b0d24d922f017068 I1121 10:21:25.389967 5400 log_verifier.cc:121] Checking tablet 787130b10eda4a71b2212bdb7ed2f140 I1121 10:21:25.467787 5400 log_verifier.cc:172] Verified matching terms for 240 ops in tablet 787130b10eda4a71b2212bdb7ed2f140 I1121 10:21:25.467840 5400 log_verifier.cc:121] Checking tablet 7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:25.550194 5400 log_verifier.cc:172] Verified matching terms for 295 ops in tablet 7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:25.550251 5400 log_verifier.cc:121] Checking tablet 7d59736e513f426095bedf84bbeec48d I1121 10:21:25.627235 5400 log_verifier.cc:172] Verified matching terms for 256 ops in tablet 7d59736e513f426095bedf84bbeec48d I1121 10:21:25.627287 5400 log_verifier.cc:121] Checking tablet 865e88694908474197879cd8d34789ad I1121 10:21:25.714470 5400 log_verifier.cc:172] Verified matching terms for 361 ops in tablet 865e88694908474197879cd8d34789ad I1121 10:21:25.714535 5400 log_verifier.cc:121] Checking tablet 993b9cf62f6c4e6296acbb32382ab037 I1121 10:21:25.797165 5400 log_verifier.cc:172] Verified matching terms for 288 ops in tablet 993b9cf62f6c4e6296acbb32382ab037 I1121 10:21:25.797224 5400 log_verifier.cc:121] Checking tablet b0cf374217f64043803515e5125b6de0 I1121 10:21:25.854729 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet b0cf374217f64043803515e5125b6de0 I1121 10:21:25.854759 5400 log_verifier.cc:121] Checking tablet c2bd898e6d3340f5982c1386e0ebd41a I1121 10:21:25.940057 5400 log_verifier.cc:172] Verified matching terms for 323 ops in tablet c2bd898e6d3340f5982c1386e0ebd41a I1121 10:21:25.940119 5400 log_verifier.cc:121] Checking tablet c5abf7a6c891418482e200e7115e2c2d I1121 10:21:26.016769 5400 log_verifier.cc:172] Verified matching terms for 239 ops in tablet c5abf7a6c891418482e200e7115e2c2d I1121 10:21:26.016841 5400 log_verifier.cc:121] Checking tablet d84faff8cd0f433d8de8664fd0054d2d I1121 10:21:26.099633 5400 log_verifier.cc:172] Verified matching terms for 292 ops in tablet d84faff8cd0f433d8de8664fd0054d2d I1121 10:21:26.099691 5400 log_verifier.cc:121] Checking tablet f30bc779dfa840c6ae504b94d82eaa5a I1121 10:21:26.175972 5400 log_verifier.cc:172] Verified matching terms for 250 ops in tablet f30bc779dfa840c6ae504b94d82eaa5a I1121 10:21:26.176290 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data --block_manager=log --rpc_bind_addresses=127.5.70.8:0 --local_ip_for_outbound_sockets=127.5.70.8 --webserver_interface=127.5.70.8 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:44133 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:26.224050 9576 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:26.224222 9576 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:26.225163 9576 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:26.225184 9576 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:26.225203 9576 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.8 I1121 10:21:26.226502 9576 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:44133 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.8:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb --webserver_interface=127.5.70.8 --webserver_port=0 --tserver_master_addrs=127.5.70.62:38839 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.9576 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.8 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:26.229001 9576 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:26.230196 9576 hybrid_clock.cc:183] HybridClock initialized: now 1574331686230081 us; error 116 us; skew 500 ppm I1121 10:21:26.230365 9576 fs_manager.cc:261] Metadata directory not provided I1121 10:21:26.230398 9576 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:26.230516 9576 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:26.230540 9576 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:26.231674 9576 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/instance: uuid: "51f14034c89344f79f40de55c5e1cfc9" format_stamp: "Formatted at 2019-11-21 10:21:26 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:26.231931 9576 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal/instance: uuid: "51f14034c89344f79f40de55c5e1cfc9" format_stamp: "Formatted at 2019-11-21 10:21:26 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:26.233470 9576 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s I1121 10:21:26.233826 9576 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:26.233963 9576 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:26.234875 9576 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:26.234911 9576 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal uuid: "51f14034c89344f79f40de55c5e1cfc9" format_stamp: "Formatted at 2019-11-21 10:21:26 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:26.234952 9576 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:26.285496 9576 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:26.285869 9576 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:26.285888 9576 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:26.294353 9576 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.8:44901 I1121 10:21:26.294723 9576 webserver.cc:382] Webserver started at http://127.5.70.8:39021/ using document root and password file I1121 10:21:26.294987 9576 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb I1121 10:21:26.300079 9697 heartbeater.cc:322] Connected to a master server at 127.5.70.62:38839 I1121 10:21:26.300137 9697 heartbeater.cc:406] Registering TS with master... I1121 10:21:26.300333 9697 heartbeater.cc:455] Master 127.5.70.62:38839 requested a full tablet report, sending... I1121 10:21:26.300709 7275 ts_manager.cc:192] Registered new tserver with Master: 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901) I1121 10:21:26.301261 7275 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.8:56529 I1121 10:21:26.301375 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 9576 I1121 10:21:26.459846 7647 consensus_queue.cc:208] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 341, Committed index: 341, Last appended: 1.341, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.460424 8021 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 1 index: 342. (index mismatch) I1121 10:21:26.460583 7772 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 1 index: 342. (index mismatch) I1121 10:21:26.460613 7899 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 1 index: 342. (index mismatch) I1121 10:21:26.460765 9792 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s I1121 10:21:26.460893 7525 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 341. Preceding OpId from leader: term: 1 index: 342. (index mismatch) I1121 10:21:26.461141 9790 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s I1121 10:21:26.461221 8291 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s I1121 10:21:26.461292 9792 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 342, Last known committed idx: 341, Time since last communication: 0.000s I1121 10:21:26.462150 8291 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.342: config changed from index -1 to 342, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.462349 7899 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.342: config changed from index -1 to 342, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.463133 7525 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.342: config changed from index -1 to 342, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.463702 8021 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.342: config changed from index -1 to 342, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.464068 7271 catalog_manager.cc:4143] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index -1 to 342, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:26.465009 7772 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.342: config changed from index -1 to 342, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } W1121 10:21:26.466560 7584 consensus_peers.cc:473] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 169c4cf587a949748d1734c65258b7a7. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:26.469821 7899 consensus_queue.cc:208] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 270, Committed index: 270, Last appended: 1.270, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.470373 7647 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch) I1121 10:21:26.470607 8659 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch) I1121 10:21:26.470649 9812 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s I1121 10:21:26.470780 8021 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch) I1121 10:21:26.470804 7402 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 270. Preceding OpId from leader: term: 1 index: 271. (index mismatch) I1121 10:21:26.470975 9812 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s I1121 10:21:26.471268 9811 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s I1121 10:21:26.471529 9810 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 271, Last known committed idx: 270, Time since last communication: 0.000s I1121 10:21:26.472306 9812 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.271: config changed from index 176 to 271, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.472474 8021 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 176 to 271, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.472640 8659 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 176 to 271, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.473654 7275 catalog_manager.cc:4143] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index 176 to 271, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.473902 7402 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 176 to 271, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.474252 7647 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.271: config changed from index 176 to 271, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } W1121 10:21:26.475184 7832 consensus_peers.cc:473] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c8cdc07bf6f4480b269ca34a9b3e426. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:26.479140 7647 consensus_queue.cc:208] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 288, Committed index: 288, Last appended: 1.288, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.479660 8021 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch) I1121 10:21:26.479900 9798 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s I1121 10:21:26.481544 7525 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch) I1121 10:21:26.481513 8659 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch) W1121 10:21:26.481889 7584 consensus_peers.cc:473] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 993b9cf62f6c4e6296acbb32382ab037. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:26.481992 9792 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s I1121 10:21:26.482103 9792 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s I1121 10:21:26.482118 7772 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 288. Preceding OpId from leader: term: 1 index: 289. (index mismatch) I1121 10:21:26.482383 9798 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 289, Last known committed idx: 288, Time since last communication: 0.000s I1121 10:21:26.483065 7525 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 91 to 289, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.484212 9792 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.289: config changed from index 91 to 289, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.484956 7772 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 91 to 289, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.485129 7276 catalog_manager.cc:4143] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf reported cstate change: config changed from index 91 to 289, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.485378 8659 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 91 to 289, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.485821 8021 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.289: config changed from index 91 to 289, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.491304 7772 consensus_queue.cc:208] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.491850 7647 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:26.491991 8659 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:26.492014 7899 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:26.492151 9836 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I1121 10:21:26.492193 7525 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:26.492226 9380 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I1121 10:21:26.492321 9380 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I1121 10:21:26.492621 9837 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I1121 10:21:26.493296 9837 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.493641 7899 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.493654 8659 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.493909 7525 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.495038 7271 catalog_manager.cc:4143] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index -1 to 2, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "1b3038ad6c564f438aa18383cddc769a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.496142 7647 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } W1121 10:21:26.497241 7708 consensus_peers.cc:473] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0cf374217f64043803515e5125b6de0. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:26.501797 7402 consensus_queue.cc:208] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 1.250, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.502588 7772 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch) I1121 10:21:26.502609 8659 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch) I1121 10:21:26.502678 7899 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch) I1121 10:21:26.502717 7525 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch) I1121 10:21:26.502823 9262 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s I1121 10:21:26.503006 9850 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s I1121 10:21:26.503166 9851 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s I1121 10:21:26.503311 9852 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s I1121 10:21:26.504269 9852 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.251: config changed from index 73 to 251, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.504741 7899 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 73 to 251, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.505839 7772 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 73 to 251, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.507019 8659 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 73 to 251, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:26.507074 7274 catalog_manager.cc:4143] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 73 to 251, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:26.507076 7525 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 73 to 251, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } W1121 10:21:26.508314 7334 consensus_peers.cc:473] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f30bc779dfa840c6ae504b94d82eaa5a. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:26.872833 9908 ts_tablet_manager.cc:721] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:26.872897 9908 tablet_copy_client.cc:243] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:43909 I1121 10:21:26.875674 7671 tablet_copy_service.cc:135] P f5bc6d2f9c574b4db9396f9f900c1141: Received BeginTabletCopySession request for tablet 993b9cf62f6c4e6296acbb32382ab037 from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:52885) I1121 10:21:26.875758 7671 tablet_copy_service.cc:156] P f5bc6d2f9c574b4db9396f9f900c1141: Beginning new tablet copy session on tablet 993b9cf62f6c4e6296acbb32382ab037 from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:52885: session id = 51f14034c89344f79f40de55c5e1cfc9-993b9cf62f6c4e6296acbb32382ab037 I1121 10:21:26.876312 7671 tablet_copy_source_session.cc:204] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:26.877068 9908 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 993b9cf62f6c4e6296acbb32382ab037. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:26.879858 9908 tablet_copy_client.cc:581] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:26.879969 9908 tablet_copy_client.cc:544] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:26.881517 9908 tablet_copy_client.cc:420] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:26.882138 9908 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:26.885833 9908 log.cc:810] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: Log is configured to *not* fsync() on all Append() calls I1121 10:21:26.890285 9911 ts_tablet_manager.cc:721] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:26.890347 9911 tablet_copy_client.cc:243] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.5:36445 I1121 10:21:26.893657 7919 tablet_copy_service.cc:135] P 81d8bacc3ca64d628a255ad416a77c3d: Received BeginTabletCopySession request for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:52957) I1121 10:21:26.893739 7919 tablet_copy_service.cc:156] P 81d8bacc3ca64d628a255ad416a77c3d: Beginning new tablet copy session on tablet 5c8cdc07bf6f4480b269ca34a9b3e426 from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:52957: session id = 51f14034c89344f79f40de55c5e1cfc9-5c8cdc07bf6f4480b269ca34a9b3e426 I1121 10:21:26.894197 7919 tablet_copy_source_session.cc:204] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:26.894630 9911 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5c8cdc07bf6f4480b269ca34a9b3e426. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:26.895689 9911 tablet_copy_client.cc:581] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:26.895805 9911 tablet_copy_client.cc:544] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:26.898243 9911 tablet_copy_client.cc:420] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:26.898887 9911 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:26.909806 9908 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=289 overwritten=0 applied=289 ignored=0} inserts{seen=983 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:26.910085 9908 tablet_bootstrap.cc:439] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:26.910248 9908 ts_tablet_manager.cc:1104] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.006s I1121 10:21:26.911221 9908 raft_consensus.cc:337] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.911279 9908 raft_consensus.cc:702] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:26.911372 9908 consensus_queue.cc:231] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 289, Last appended: 1.289, Last appended by leader: 289, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 289 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.911556 9908 ts_tablet_manager.cc:1132] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s I1121 10:21:26.911556 9697 heartbeater.cc:447] Master 127.5.70.62:38839 was elected leader, sending a full tablet report... I1121 10:21:26.911912 7671 tablet_copy_service.cc:325] P f5bc6d2f9c574b4db9396f9f900c1141: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-993b9cf62f6c4e6296acbb32382ab037 received from {username='slave'} at 127.5.70.8:52885 I1121 10:21:26.911939 7671 tablet_copy_service.cc:413] P f5bc6d2f9c574b4db9396f9f900c1141: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-993b9cf62f6c4e6296acbb32382ab037 on tablet 993b9cf62f6c4e6296acbb32382ab037 with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:26.917197 9908 ts_tablet_manager.cc:721] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909) I1121 10:21:26.917240 9908 tablet_copy_client.cc:243] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:43909 I1121 10:21:26.917428 7671 tablet_copy_service.cc:135] P f5bc6d2f9c574b4db9396f9f900c1141: Received BeginTabletCopySession request for tablet 169c4cf587a949748d1734c65258b7a7 from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:52885) I1121 10:21:26.917459 7671 tablet_copy_service.cc:156] P f5bc6d2f9c574b4db9396f9f900c1141: Beginning new tablet copy session on tablet 169c4cf587a949748d1734c65258b7a7 from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:52885: session id = 51f14034c89344f79f40de55c5e1cfc9-169c4cf587a949748d1734c65258b7a7 I1121 10:21:26.917969 7671 tablet_copy_source_session.cc:204] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:26.918313 9908 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 169c4cf587a949748d1734c65258b7a7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:26.919131 9908 tablet_copy_client.cc:581] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:26.919247 9908 tablet_copy_client.cc:544] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:26.921473 9908 tablet_copy_client.cc:420] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:26.922120 9908 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:26.923096 9911 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=271 overwritten=0 applied=271 ignored=0} inserts{seen=887 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:26.923280 9911 tablet_bootstrap.cc:439] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:26.923393 9911 ts_tablet_manager.cc:1104] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.025s user 0.024s sys 0.000s I1121 10:21:26.923710 9911 raft_consensus.cc:337] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.923743 9911 raft_consensus.cc:702] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:26.923804 9911 consensus_queue.cc:231] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 271, Last appended: 1.271, Last appended by leader: 271, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 271 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.923889 9911 ts_tablet_manager.cc:1132] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:21:26.924340 7919 tablet_copy_service.cc:325] P 81d8bacc3ca64d628a255ad416a77c3d: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-5c8cdc07bf6f4480b269ca34a9b3e426 received from {username='slave'} at 127.5.70.8:52957 I1121 10:21:26.924394 7919 tablet_copy_service.cc:413] P 81d8bacc3ca64d628a255ad416a77c3d: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-5c8cdc07bf6f4480b269ca34a9b3e426 on tablet 5c8cdc07bf6f4480b269ca34a9b3e426 with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:26.950700 9908 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=342 overwritten=0 applied=342 ignored=0} inserts{seen=961 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:26.950897 9908 tablet_bootstrap.cc:439] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:26.951014 9908 ts_tablet_manager.cc:1104] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.029s user 0.024s sys 0.004s I1121 10:21:26.951136 9908 raft_consensus.cc:337] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.951156 9908 raft_consensus.cc:702] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:26.951192 9908 consensus_queue.cc:231] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 342, Last appended: 1.342, Last appended by leader: 342, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 342 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:26.951269 9908 ts_tablet_manager.cc:1132] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:26.951495 7671 tablet_copy_service.cc:325] P f5bc6d2f9c574b4db9396f9f900c1141: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-169c4cf587a949748d1734c65258b7a7 received from {username='slave'} at 127.5.70.8:52885 I1121 10:21:26.951521 7671 tablet_copy_service.cc:413] P f5bc6d2f9c574b4db9396f9f900c1141: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-169c4cf587a949748d1734c65258b7a7 on tablet 169c4cf587a949748d1734c65258b7a7 with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:27.011831 9908 ts_tablet_manager.cc:721] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995) I1121 10:21:27.011927 9908 tablet_copy_client.cc:243] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:36995 I1121 10:21:27.014509 7422 tablet_copy_service.cc:135] P 63b9dc8cf49b47ebb644515af2f427e7: Received BeginTabletCopySession request for tablet f30bc779dfa840c6ae504b94d82eaa5a from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:55719) I1121 10:21:27.014575 7422 tablet_copy_service.cc:156] P 63b9dc8cf49b47ebb644515af2f427e7: Beginning new tablet copy session on tablet f30bc779dfa840c6ae504b94d82eaa5a from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:55719: session id = 51f14034c89344f79f40de55c5e1cfc9-f30bc779dfa840c6ae504b94d82eaa5a I1121 10:21:27.015034 7422 tablet_copy_source_session.cc:204] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:27.015493 9908 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet f30bc779dfa840c6ae504b94d82eaa5a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:27.016497 9908 tablet_copy_client.cc:581] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:27.016733 9908 tablet_copy_client.cc:544] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:27.018503 9908 tablet_copy_client.cc:420] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:27.019215 9908 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:27.038300 9908 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=251 overwritten=0 applied=251 ignored=0} inserts{seen=920 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:27.038518 9908 tablet_bootstrap.cc:439] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:27.038642 9908 ts_tablet_manager.cc:1104] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.002s I1121 10:21:27.038825 9908 raft_consensus.cc:337] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:27.038851 9908 raft_consensus.cc:702] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:27.038915 9908 consensus_queue.cc:231] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 251, Last appended: 1.251, Last appended by leader: 251, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:27.038992 9908 ts_tablet_manager.cc:1132] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:27.039304 7422 tablet_copy_service.cc:325] P 63b9dc8cf49b47ebb644515af2f427e7: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-f30bc779dfa840c6ae504b94d82eaa5a received from {username='slave'} at 127.5.70.8:55719 I1121 10:21:27.039332 7422 tablet_copy_service.cc:413] P 63b9dc8cf49b47ebb644515af2f427e7: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-f30bc779dfa840c6ae504b94d82eaa5a on tablet f30bc779dfa840c6ae504b94d82eaa5a with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:27.046336 9908 ts_tablet_manager.cc:721] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993) I1121 10:21:27.046384 9908 tablet_copy_client.cc:243] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:32993 I1121 10:21:27.049026 7795 tablet_copy_service.cc:135] P 1b3038ad6c564f438aa18383cddc769a: Received BeginTabletCopySession request for tablet b0cf374217f64043803515e5125b6de0 from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:36893) I1121 10:21:27.049060 7795 tablet_copy_service.cc:156] P 1b3038ad6c564f438aa18383cddc769a: Beginning new tablet copy session on tablet b0cf374217f64043803515e5125b6de0 from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:36893: session id = 51f14034c89344f79f40de55c5e1cfc9-b0cf374217f64043803515e5125b6de0 I1121 10:21:27.049723 7795 tablet_copy_source_session.cc:204] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:27.050158 9908 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b0cf374217f64043803515e5125b6de0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:27.050995 9908 tablet_copy_client.cc:581] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:27.051084 9908 tablet_copy_client.cc:544] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:27.051659 9908 tablet_copy_client.cc:420] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:27.052109 9908 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:27.053867 9908 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:27.053987 9908 tablet_bootstrap.cc:439] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:27.054049 9908 ts_tablet_manager.cc:1104] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s I1121 10:21:27.054225 9908 raft_consensus.cc:337] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:27.054248 9908 raft_consensus.cc:702] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:27.054283 9908 consensus_queue.cc:231] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:27.054419 9908 ts_tablet_manager.cc:1132] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:27.054632 7795 tablet_copy_service.cc:325] P 1b3038ad6c564f438aa18383cddc769a: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-b0cf374217f64043803515e5125b6de0 received from {username='slave'} at 127.5.70.8:36893 I1121 10:21:27.054656 7795 tablet_copy_service.cc:413] P 1b3038ad6c564f438aa18383cddc769a: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-b0cf374217f64043803515e5125b6de0 on tablet b0cf374217f64043803515e5125b6de0 with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:27.301762 9650 raft_consensus.cc:1123] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.270->[1.271-1.271] Dedup: 1.271->[] I1121 10:21:27.410806 9650 raft_consensus.cc:1123] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.288->[1.289-1.289] Dedup: 1.289->[] I1121 10:21:27.476658 9650 raft_consensus.cc:1123] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.250->[1.251-1.251] Dedup: 1.251->[] I1121 10:21:27.527549 9650 raft_consensus.cc:1123] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.341->[1.342-1.342] Dedup: 1.342->[] I1121 10:21:27.589076 9650 raft_consensus.cc:1123] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[] I1121 10:21:27.734025 9967 raft_consensus.cc:971] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:27.734268 9967 consensus_queue.cc:208] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 271, Committed index: 271, Last appended: 1.271, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:27.734860 7648 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch) I1121 10:21:27.734863 7402 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch) I1121 10:21:27.734879 8021 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch) I1121 10:21:27.734958 8659 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch) I1121 10:21:27.735091 9650 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 271. Preceding OpId from leader: term: 1 index: 272. (index mismatch) I1121 10:21:27.735096 10024 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s I1121 10:21:27.735226 9810 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s I1121 10:21:27.735317 10025 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s I1121 10:21:27.735540 10024 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s I1121 10:21:27.735694 10025 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 272, Last known committed idx: 271, Time since last communication: 0.000s I1121 10:21:27.736443 9968 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.272: config changed from index 271 to 272, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.736614 7648 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.736805 8659 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.737747 8021 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.737911 7274 catalog_manager.cc:4143] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index 271 to 272, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:27.738854 7402 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.739045 9650 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.272: config changed from index 271 to 272, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 272 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.741475 7899 consensus_queue.cc:208] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 272, Committed index: 272, Last appended: 1.272, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:27.741847 8021 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 272. Preceding OpId from leader: term: 1 index: 273. (index mismatch) I1121 10:21:27.741853 7648 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 272. Preceding OpId from leader: term: 1 index: 273. (index mismatch) I1121 10:21:27.741928 8659 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 272. Preceding OpId from leader: term: 1 index: 273. (index mismatch) I1121 10:21:27.742000 9650 raft_consensus.cc:1181] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 272. Preceding OpId from leader: term: 1 index: 273. (index mismatch) I1121 10:21:27.742033 9810 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s I1121 10:21:27.742137 9810 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s I1121 10:21:27.742332 9967 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s I1121 10:21:27.742441 10025 consensus_queue.cc:984] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 273, Last known committed idx: 272, Time since last communication: 0.000s I1121 10:21:27.743271 9810 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.273: config changed from index 272 to 273, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.743449 9650 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.273: config changed from index 272 to 273, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.743466 8659 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.273: config changed from index 272 to 273, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.744160 7648 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.273: config changed from index 272 to 273, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.744235 8021 raft_consensus.cc:2801] T 5c8cdc07bf6f4480b269ca34a9b3e426 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.273: config changed from index 272 to 273, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.744684 7271 catalog_manager.cc:4143] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 51f14034c89344f79f40de55c5e1cfc9 reported cstate change: config changed from index 272 to 273, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 273 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.745411 7370 tablet_service.cc:1067] Processing DeleteTablet for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 with delete_type TABLET_DATA_TOMBSTONED (TS 63b9dc8cf49b47ebb644515af2f427e7 not found in new config with opid_index 273) from {username='slave'} at 127.0.0.1:34708 I1121 10:21:27.745564 10037 tablet_replica.cc:282] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7: stopping tablet replica I1121 10:21:27.745600 10037 raft_consensus.cc:2156] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:27.745693 10037 raft_consensus.cc:2183] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:27.745795 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 5c8cdc07bf6f4480b269ca34a9b3e426 with cas_config_opid_index 272: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:27.746767 10037 ts_tablet_manager.cc:1457] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:27.748196 10037 ts_tablet_manager.cc:1470] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.272 I1121 10:21:27.748272 10037 log.cc:1153] T 5c8cdc07bf6f4480b269ca34a9b3e426 P 63b9dc8cf49b47ebb644515af2f427e7: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/5c8cdc07bf6f4480b269ca34a9b3e426 I1121 10:21:27.748811 7262 catalog_manager.cc:3512] TS 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995): tablet 5c8cdc07bf6f4480b269ca34a9b3e426 (table rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]) successfully deleted I1121 10:21:27.829905 9790 raft_consensus.cc:971] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:27.830168 9790 consensus_queue.cc:208] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 1.289, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:27.830691 8021 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch) I1121 10:21:27.830828 8659 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch) I1121 10:21:27.830878 9650 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch) I1121 10:21:27.830909 7772 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch) I1121 10:21:27.831032 7525 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch) I1121 10:21:27.831060 9996 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s I1121 10:21:27.831336 9996 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s I1121 10:21:27.831452 10060 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s I1121 10:21:27.831542 9996 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s I1121 10:21:27.831634 9994 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s I1121 10:21:27.832635 9994 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.290: config changed from index 289 to 290, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.832954 8659 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 289 to 290, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.834290 8021 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 289 to 290, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.834408 7275 catalog_manager.cc:4143] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index 289 to 290, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:27.834692 7772 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 289 to 290, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.836086 7525 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 289 to 290, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.836148 9650 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 289 to 290, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.836592 7648 consensus_queue.cc:208] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 1.290, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:27.837013 8659 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 290. Preceding OpId from leader: term: 1 index: 291. (index mismatch) I1121 10:21:27.837096 7772 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 290. Preceding OpId from leader: term: 1 index: 291. (index mismatch) I1121 10:21:27.837221 10059 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s I1121 10:21:27.837254 7525 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 290. Preceding OpId from leader: term: 1 index: 291. (index mismatch) I1121 10:21:27.837301 10059 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s I1121 10:21:27.837842 10071 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s I1121 10:21:27.838048 9650 raft_consensus.cc:1181] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 290. Preceding OpId from leader: term: 1 index: 291. (index mismatch) I1121 10:21:27.838426 10071 consensus_queue.cc:984] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s I1121 10:21:27.838636 9996 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.291: config changed from index 290 to 291, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.838975 7525 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.291: config changed from index 290 to 291, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.839294 9650 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.291: config changed from index 290 to 291, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.839680 8659 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.291: config changed from index 290 to 291, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.839912 7261 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 993b9cf62f6c4e6296acbb32382ab037 with cas_config_opid_index 290: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:27.840143 7275 catalog_manager.cc:4143] T 993b9cf62f6c4e6296acbb32382ab037 P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index 290 to 291, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:27.841068 8003 tablet_service.cc:1067] Processing DeleteTablet for tablet 993b9cf62f6c4e6296acbb32382ab037 with delete_type TABLET_DATA_TOMBSTONED (TS c9e9c35ff1cb4b68ba7cf5c037a98347 not found in new config with opid_index 291) from {username='slave'} at 127.0.0.1:44220 I1121 10:21:27.841377 7772 raft_consensus.cc:2801] T 993b9cf62f6c4e6296acbb32382ab037 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.291: config changed from index 290 to 291, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.842655 10077 tablet_replica.cc:282] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347: stopping tablet replica I1121 10:21:27.842782 10077 raft_consensus.cc:2156] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:27.842797 10077 raft_consensus.cc:2183] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:27.843763 10077 ts_tablet_manager.cc:1457] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:27.844285 7402 consensus_queue.cc:208] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 300, Committed index: 300, Last appended: 1.300, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:27.844820 8021 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch) I1121 10:21:27.844995 7648 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch) I1121 10:21:27.845002 7899 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch) W1121 10:21:27.845191 7334 consensus_peers.cc:473] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a9a3a26473340c381d502b8e9e1660c. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:27.845227 8659 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 300. Preceding OpId from leader: term: 1 index: 301. (index mismatch) I1121 10:21:27.845240 9262 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s I1121 10:21:27.845316 9262 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s I1121 10:21:27.845376 9995 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s I1121 10:21:27.845474 9995 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 301, Last known committed idx: 300, Time since last communication: 0.000s I1121 10:21:27.846290 10077 ts_tablet_manager.cc:1470] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.290 I1121 10:21:27.846334 10077 log.cc:1153] T 993b9cf62f6c4e6296acbb32382ab037 P c9e9c35ff1cb4b68ba7cf5c037a98347: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal/wals/993b9cf62f6c4e6296acbb32382ab037 I1121 10:21:27.846386 9262 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.301: config changed from index 167 to 301, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.846765 7262 catalog_manager.cc:3512] TS c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833): tablet 993b9cf62f6c4e6296acbb32382ab037 (table rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]) successfully deleted I1121 10:21:27.846765 7648 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 167 to 301, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.846984 8021 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 167 to 301, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.847931 7271 catalog_manager.cc:4143] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index 167 to 301, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.847960 7899 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 167 to 301, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.849261 8659 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.301: config changed from index 167 to 301, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.865890 9995 raft_consensus.cc:971] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:27.866137 9995 consensus_queue.cc:208] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 1.251, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:27.866876 7525 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch) I1121 10:21:27.866955 8660 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch) I1121 10:21:27.866986 7899 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch) I1121 10:21:27.867127 9995 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s I1121 10:21:27.867120 7772 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch) I1121 10:21:27.867218 9995 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s I1121 10:21:27.867344 10116 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s I1121 10:21:27.867406 9650 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch) I1121 10:21:27.867569 9262 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s I1121 10:21:27.867717 9995 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s I1121 10:21:27.869134 9262 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.252: config changed from index 251 to 252, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.869503 8660 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 251 to 252, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.869549 7772 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 251 to 252, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.870477 7402 consensus_queue.cc:208] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.256, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:27.870635 7273 catalog_manager.cc:4143] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 251 to 252, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:27.871134 7899 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 251 to 252, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.871191 8021 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch) I1121 10:21:27.871392 9262 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s I1121 10:21:27.872216 7899 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch) I1121 10:21:27.872246 8660 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch) I1121 10:21:27.872248 7772 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch) W1121 10:21:27.872287 7334 consensus_peers.cc:473] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d59736e513f426095bedf84bbeec48d. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:27.872445 9995 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s I1121 10:21:27.872505 7525 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 251 to 252, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.872576 9262 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s I1121 10:21:27.872682 9262 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s I1121 10:21:27.873956 9650 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 251 to 252, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { replace: true } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.874280 7402 consensus_queue.cc:208] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:27.874711 10116 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.257: config changed from index 68 to 257, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.874868 8660 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch) I1121 10:21:27.874984 7899 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch) I1121 10:21:27.875046 9654 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch) I1121 10:21:27.875111 10084 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s I1121 10:21:27.875142 7772 raft_consensus.cc:1181] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch) I1121 10:21:27.875309 9995 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s I1121 10:21:27.875406 10084 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s I1121 10:21:27.875543 9262 consensus_queue.cc:984] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s I1121 10:21:27.875589 7899 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 68 to 257, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.876168 10084 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index 252 to 253, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.876277 7893 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 252 to 253, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.876595 9654 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 252 to 253, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.877049 8659 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 68 to 257, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.877224 7262 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet f30bc779dfa840c6ae504b94d82eaa5a with cas_config_opid_index 252: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:27.877355 7278 catalog_manager.cc:4143] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index 68 to 257, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.877647 7271 catalog_manager.cc:4143] T f30bc779dfa840c6ae504b94d82eaa5a P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 252 to 253, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:27.877692 7772 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 252 to 253, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.878394 7501 tablet_service.cc:1067] Processing DeleteTablet for tablet f30bc779dfa840c6ae504b94d82eaa5a with delete_type TABLET_DATA_TOMBSTONED (TS 41118f5c2e8f48179ffbebbfb6da2edf not found in new config with opid_index 253) from {username='slave'} at 127.0.0.1:49436 I1121 10:21:27.878747 8660 raft_consensus.cc:2801] T f30bc779dfa840c6ae504b94d82eaa5a P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 252 to 253, VOTER 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:27.879379 10128 tablet_replica.cc:282] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf: stopping tablet replica I1121 10:21:27.879639 10128 raft_consensus.cc:2156] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:27.879776 10128 raft_consensus.cc:2183] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:27.880919 10128 ts_tablet_manager.cc:1457] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:27.881734 8021 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 68 to 257, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.881845 7772 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index 68 to 257, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.882469 10128 ts_tablet_manager.cc:1470] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.252 I1121 10:21:27.882550 10128 log.cc:1153] T f30bc779dfa840c6ae504b94d82eaa5a P 41118f5c2e8f48179ffbebbfb6da2edf: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/f30bc779dfa840c6ae504b94d82eaa5a I1121 10:21:27.883030 7263 catalog_manager.cc:3512] TS 41118f5c2e8f48179ffbebbfb6da2edf (127.5.70.2:39291): tablet f30bc779dfa840c6ae504b94d82eaa5a (table rebalance_test_table_4 [id=692cf724e38648ed81e17afafb0bbf35]) successfully deleted I1121 10:21:27.900810 7893 consensus_queue.cc:208] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 239, Committed index: 239, Last appended: 1.239, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:27.901369 7525 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch) I1121 10:21:27.901396 7402 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch) I1121 10:21:27.901396 8021 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch) I1121 10:21:27.901484 7648 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 239. Preceding OpId from leader: term: 1 index: 240. (index mismatch) I1121 10:21:27.901631 9967 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s I1121 10:21:27.901738 9810 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s W1121 10:21:27.901770 7832 consensus_peers.cc:473] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5abf7a6c891418482e200e7115e2c2d. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:27.901825 10024 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s I1121 10:21:27.901953 9810 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 240, Last known committed idx: 239, Time since last communication: 0.000s I1121 10:21:27.902706 10024 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.240: config changed from index 62 to 240, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.902881 7648 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 62 to 240, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.902992 7402 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 62 to 240, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.903055 8021 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 62 to 240, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.904281 7525 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.240: config changed from index 62 to 240, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:27.904299 7278 catalog_manager.cc:4143] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index 62 to 240, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.023010 10060 raft_consensus.cc:971] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:28.023277 10060 consensus_queue.cc:208] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 342, Committed index: 342, Last appended: 1.342, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:28.023931 9654 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 342. Preceding OpId from leader: term: 1 index: 343. (index mismatch) I1121 10:21:28.023944 7772 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 342. Preceding OpId from leader: term: 1 index: 343. (index mismatch) I1121 10:21:28.023967 7525 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 342. Preceding OpId from leader: term: 1 index: 343. (index mismatch) I1121 10:21:28.024108 7893 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 342. Preceding OpId from leader: term: 1 index: 343. (index mismatch) I1121 10:21:28.024267 8021 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 342. Preceding OpId from leader: term: 1 index: 343. (index mismatch) I1121 10:21:28.024291 9994 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s I1121 10:21:28.024370 10071 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s I1121 10:21:28.024466 10059 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s I1121 10:21:28.024574 10071 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s I1121 10:21:28.025027 10059 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 343, Last known committed idx: 342, Time since last communication: 0.000s I1121 10:21:28.026283 9790 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.343: config changed from index 342 to 343, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.026387 8021 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.343: config changed from index 342 to 343, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.026494 7893 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.343: config changed from index 342 to 343, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.027674 9654 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.343: config changed from index 342 to 343, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.027760 7271 catalog_manager.cc:4143] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index 342 to 343, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.028043 7525 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.343: config changed from index 342 to 343, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.028342 7772 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.343: config changed from index 342 to 343, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 343 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.029942 7648 consensus_queue.cc:208] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 343, Committed index: 343, Last appended: 1.343, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:28.030331 8021 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch) I1121 10:21:28.030356 9654 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch) I1121 10:21:28.030372 7893 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch) I1121 10:21:28.030372 7525 raft_consensus.cc:1181] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer f5bc6d2f9c574b4db9396f9f900c1141: Log matching property violated. Preceding OpId in replica: term: 1 index: 343. Preceding OpId from leader: term: 1 index: 344. (index mismatch) I1121 10:21:28.030529 10060 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s I1121 10:21:28.030596 10059 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s I1121 10:21:28.030695 9994 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s I1121 10:21:28.030758 10071 consensus_queue.cc:984] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 344, Last known committed idx: 343, Time since last communication: 0.000s I1121 10:21:28.031646 10071 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 LEADER]: Committing config change with OpId 1.344: config changed from index 343 to 344, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.031803 7525 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index 343 to 344, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.031878 9654 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index 343 to 344, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.032709 7261 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 169c4cf587a949748d1734c65258b7a7 with cas_config_opid_index 343: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:28.032856 7893 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index 343 to 344, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.032929 7275 catalog_manager.cc:4143] T 169c4cf587a949748d1734c65258b7a7 P 41118f5c2e8f48179ffbebbfb6da2edf reported cstate change: config changed from index 343 to 344, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" committed_config { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.033650 7754 tablet_service.cc:1067] Processing DeleteTablet for tablet 169c4cf587a949748d1734c65258b7a7 with delete_type TABLET_DATA_TOMBSTONED (TS 1b3038ad6c564f438aa18383cddc769a not found in new config with opid_index 344) from {username='slave'} at 127.0.0.1:42904 I1121 10:21:28.033823 10195 tablet_replica.cc:282] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a: stopping tablet replica I1121 10:21:28.033861 8021 raft_consensus.cc:2801] T 169c4cf587a949748d1734c65258b7a7 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.344: config changed from index 343 to 344, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 344 OBSOLETE_local: false peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.033883 10195 raft_consensus.cc:2156] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:28.033906 10195 raft_consensus.cc:2183] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:28.034708 10195 ts_tablet_manager.cc:1457] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:28.036307 10195 ts_tablet_manager.cc:1470] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.343 I1121 10:21:28.036347 10195 log.cc:1153] T 169c4cf587a949748d1734c65258b7a7 P 1b3038ad6c564f438aa18383cddc769a: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/169c4cf587a949748d1734c65258b7a7 I1121 10:21:28.036898 7263 catalog_manager.cc:3512] TS 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993): tablet 169c4cf587a949748d1734c65258b7a7 (table rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]) successfully deleted I1121 10:21:28.143594 7893 consensus_queue.cc:208] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.144227 8021 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch) I1121 10:21:28.144228 7772 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch) I1121 10:21:28.144325 8660 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch) I1121 10:21:28.144353 7402 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 1 index: 282. (index mismatch) I1121 10:21:28.144596 9968 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s W1121 10:21:28.144655 7832 consensus_peers.cc:473] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 605a58ce874c4170b0d24d922f017068. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:28.144685 9968 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s I1121 10:21:28.144817 9967 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s I1121 10:21:28.145097 10025 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s I1121 10:21:28.145975 10025 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.282: config changed from index 146 to 282, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.146198 8021 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 146 to 282, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.147228 7402 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 146 to 282, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.147269 7275 catalog_manager.cc:4143] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index 146 to 282, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.147421 7772 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 146 to 282, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.148720 8660 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.282: config changed from index 146 to 282, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.198933 10256 raft_consensus.cc:971] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:28.199265 10256 consensus_queue.cc:208] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:28.199934 9654 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:28.200099 7525 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:28.200132 7893 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:28.200139 7648 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:28.200464 10259 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I1121 10:21:28.200467 8660 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:28.200575 9843 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I1121 10:21:28.200680 10260 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I1121 10:21:28.200780 10258 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I1121 10:21:28.201226 10256 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I1121 10:21:28.201930 10256 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.202085 8660 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.202375 7648 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.203335 7273 catalog_manager.cc:4143] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 2 to 3, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b3038ad6c564f438aa18383cddc769a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.203492 7893 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.203838 9654 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.205099 7525 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } attrs { replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } W1121 10:21:28.206115 10256 consensus_queue.cc:1099] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } } I1121 10:21:28.206300 7772 consensus_queue.cc:208] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:28.206637 7893 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:28.206674 9654 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:28.206693 8660 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:28.206720 7525 raft_consensus.cc:1181] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 1b3038ad6c564f438aa18383cddc769a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:28.206912 10259 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:28.206992 10261 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:28.207106 9843 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:28.207176 10260 consensus_queue.cc:984] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:28.207857 9843 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 1b3038ad6c564f438aa18383cddc769a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.207993 7893 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.208041 9654 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.208079 8660 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.208773 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet b0cf374217f64043803515e5125b6de0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:28.208981 7271 catalog_manager.cc:4143] T b0cf374217f64043803515e5125b6de0 P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index 3 to 4, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "1b3038ad6c564f438aa18383cddc769a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.209226 7525 raft_consensus.cc:2801] T b0cf374217f64043803515e5125b6de0 P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:28.209614 7628 tablet_service.cc:1067] Processing DeleteTablet for tablet b0cf374217f64043803515e5125b6de0 with delete_type TABLET_DATA_TOMBSTONED (TS f5bc6d2f9c574b4db9396f9f900c1141 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39024 I1121 10:21:28.209774 10267 tablet_replica.cc:282] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: stopping tablet replica I1121 10:21:28.209823 10267 raft_consensus.cc:2156] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:28.209841 10267 raft_consensus.cc:2183] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:28.210752 10267 ts_tablet_manager.cc:1457] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:28.212195 10267 ts_tablet_manager.cc:1470] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3 I1121 10:21:28.212235 10267 log.cc:1153] T b0cf374217f64043803515e5125b6de0 P f5bc6d2f9c574b4db9396f9f900c1141: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/b0cf374217f64043803515e5125b6de0 I1121 10:21:28.212677 7261 catalog_manager.cc:3512] TS f5bc6d2f9c574b4db9396f9f900c1141 (127.5.70.3:43909): tablet b0cf374217f64043803515e5125b6de0 (table TestTable [id=5c657504639147f78690abf942e8f1b9]) successfully deleted I1121 10:21:28.282172 10268 ts_tablet_manager.cc:721] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995) I1121 10:21:28.282270 10268 tablet_copy_client.cc:243] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:36995 I1121 10:21:28.282480 7422 tablet_copy_service.cc:135] P 63b9dc8cf49b47ebb644515af2f427e7: Received BeginTabletCopySession request for tablet 7d59736e513f426095bedf84bbeec48d from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:55719) I1121 10:21:28.282514 7422 tablet_copy_service.cc:156] P 63b9dc8cf49b47ebb644515af2f427e7: Beginning new tablet copy session on tablet 7d59736e513f426095bedf84bbeec48d from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:55719: session id = 51f14034c89344f79f40de55c5e1cfc9-7d59736e513f426095bedf84bbeec48d I1121 10:21:28.283008 7422 tablet_copy_source_session.cc:204] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:28.283378 10268 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7d59736e513f426095bedf84bbeec48d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:28.284292 10268 tablet_copy_client.cc:581] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:28.284368 10268 tablet_copy_client.cc:544] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:28.285182 10268 tablet_copy_client.cc:420] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:28.285702 10268 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:28.304924 10268 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=257 overwritten=0 applied=257 ignored=0} inserts{seen=951 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:28.305135 10268 tablet_bootstrap.cc:439] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:28.305251 10268 ts_tablet_manager.cc:1104] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s I1121 10:21:28.305407 10268 raft_consensus.cc:337] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.305425 10268 raft_consensus.cc:702] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:28.305475 10268 consensus_queue.cc:231] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 257, Last appended: 1.257, Last appended by leader: 257, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.305596 10268 ts_tablet_manager.cc:1132] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:28.305977 7422 tablet_copy_service.cc:325] P 63b9dc8cf49b47ebb644515af2f427e7: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-7d59736e513f426095bedf84bbeec48d received from {username='slave'} at 127.5.70.8:55719 I1121 10:21:28.306010 7422 tablet_copy_service.cc:413] P 63b9dc8cf49b47ebb644515af2f427e7: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-7d59736e513f426095bedf84bbeec48d on tablet 7d59736e513f426095bedf84bbeec48d with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:28.325636 10268 ts_tablet_manager.cc:721] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995) I1121 10:21:28.325697 10268 tablet_copy_client.cc:243] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:36995 I1121 10:21:28.326010 7422 tablet_copy_service.cc:135] P 63b9dc8cf49b47ebb644515af2f427e7: Received BeginTabletCopySession request for tablet 5a9a3a26473340c381d502b8e9e1660c from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:55719) I1121 10:21:28.326046 7422 tablet_copy_service.cc:156] P 63b9dc8cf49b47ebb644515af2f427e7: Beginning new tablet copy session on tablet 5a9a3a26473340c381d502b8e9e1660c from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:55719: session id = 51f14034c89344f79f40de55c5e1cfc9-5a9a3a26473340c381d502b8e9e1660c I1121 10:21:28.326566 7422 tablet_copy_source_session.cc:204] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:28.327031 10268 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5a9a3a26473340c381d502b8e9e1660c. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:28.327986 10268 tablet_copy_client.cc:581] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:28.328079 10268 tablet_copy_client.cc:544] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:28.329257 10268 tablet_copy_client.cc:420] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:28.329711 10268 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:28.350777 10268 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=301 overwritten=0 applied=301 ignored=0} inserts{seen=917 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:28.350987 10268 tablet_bootstrap.cc:439] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:28.351117 10268 ts_tablet_manager.cc:1104] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s I1121 10:21:28.351297 10268 raft_consensus.cc:337] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.351320 10268 raft_consensus.cc:702] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:28.351379 10268 consensus_queue.cc:231] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 301, Last appended: 1.301, Last appended by leader: 301, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 301 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.351461 10268 ts_tablet_manager.cc:1132] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:28.351706 7422 tablet_copy_service.cc:325] P 63b9dc8cf49b47ebb644515af2f427e7: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-5a9a3a26473340c381d502b8e9e1660c received from {username='slave'} at 127.5.70.8:55719 I1121 10:21:28.351735 7422 tablet_copy_service.cc:413] P 63b9dc8cf49b47ebb644515af2f427e7: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-5a9a3a26473340c381d502b8e9e1660c on tablet 5a9a3a26473340c381d502b8e9e1660c with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:28.384909 7893 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.385560 8021 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch) I1121 10:21:28.385562 7772 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch) I1121 10:21:28.385643 8660 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch) I1121 10:21:28.385710 7402 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch) W1121 10:21:28.385738 7832 consensus_peers.cc:473] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8:44901): Couldn't send request to peer 51f14034c89344f79f40de55c5e1cfc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cd3c7862a7c420cb0b87bd87a770d84. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:28.385828 10025 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s I1121 10:21:28.385936 9967 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s I1121 10:21:28.385991 9968 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s I1121 10:21:28.386059 10025 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s I1121 10:21:28.386912 10025 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 236 to 296, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.387107 7402 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 236 to 296, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.387274 8021 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 236 to 296, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.388334 7278 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index 236 to 296, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.388417 7772 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 236 to 296, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.388783 8660 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 236 to 296, NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) added. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } } I1121 10:21:28.409869 10268 ts_tablet_manager.cc:721] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:28.409914 10268 tablet_copy_client.cc:243] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.5:36445 I1121 10:21:28.410118 7919 tablet_copy_service.cc:135] P 81d8bacc3ca64d628a255ad416a77c3d: Received BeginTabletCopySession request for tablet c5abf7a6c891418482e200e7115e2c2d from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:52957) I1121 10:21:28.410145 7919 tablet_copy_service.cc:156] P 81d8bacc3ca64d628a255ad416a77c3d: Beginning new tablet copy session on tablet c5abf7a6c891418482e200e7115e2c2d from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:52957: session id = 51f14034c89344f79f40de55c5e1cfc9-c5abf7a6c891418482e200e7115e2c2d I1121 10:21:28.410645 7919 tablet_copy_source_session.cc:204] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:28.411000 10268 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c5abf7a6c891418482e200e7115e2c2d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:28.411979 10268 tablet_copy_client.cc:581] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:28.412081 10268 tablet_copy_client.cc:544] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:28.413251 10268 tablet_copy_client.cc:420] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:28.413734 10268 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:28.431013 10268 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=240 overwritten=0 applied=240 ignored=0} inserts{seen=856 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:28.431210 10268 tablet_bootstrap.cc:439] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:28.431363 10268 ts_tablet_manager.cc:1104] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s I1121 10:21:28.431566 10268 raft_consensus.cc:337] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.431587 10268 raft_consensus.cc:702] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:28.431648 10268 consensus_queue.cc:231] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 240, Last appended: 1.240, Last appended by leader: 240, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 240 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.431727 10268 ts_tablet_manager.cc:1132] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:21:28.432034 7919 tablet_copy_service.cc:325] P 81d8bacc3ca64d628a255ad416a77c3d: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-c5abf7a6c891418482e200e7115e2c2d received from {username='slave'} at 127.5.70.8:52957 I1121 10:21:28.432065 7919 tablet_copy_service.cc:413] P 81d8bacc3ca64d628a255ad416a77c3d: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-c5abf7a6c891418482e200e7115e2c2d on tablet c5abf7a6c891418482e200e7115e2c2d with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:28.764349 10268 ts_tablet_manager.cc:721] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:28.764411 10268 tablet_copy_client.cc:243] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.5:36445 I1121 10:21:28.764736 7919 tablet_copy_service.cc:135] P 81d8bacc3ca64d628a255ad416a77c3d: Received BeginTabletCopySession request for tablet 605a58ce874c4170b0d24d922f017068 from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:52957) I1121 10:21:28.764770 7919 tablet_copy_service.cc:156] P 81d8bacc3ca64d628a255ad416a77c3d: Beginning new tablet copy session on tablet 605a58ce874c4170b0d24d922f017068 from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:52957: session id = 51f14034c89344f79f40de55c5e1cfc9-605a58ce874c4170b0d24d922f017068 I1121 10:21:28.765249 7919 tablet_copy_source_session.cc:204] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:28.765621 10268 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 605a58ce874c4170b0d24d922f017068. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:28.766530 10268 tablet_copy_client.cc:581] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:28.766621 10268 tablet_copy_client.cc:544] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:28.767665 10268 tablet_copy_client.cc:420] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:28.768226 10268 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:28.790827 10268 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=282 overwritten=0 applied=282 ignored=0} inserts{seen=951 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:28.791030 10268 tablet_bootstrap.cc:439] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:28.791159 10268 ts_tablet_manager.cc:1104] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.012s I1121 10:21:28.791317 10268 raft_consensus.cc:337] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.791338 10268 raft_consensus.cc:702] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:28.791388 10268 consensus_queue.cc:231] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 282, Last appended: 1.282, Last appended by leader: 282, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 282 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.791466 10268 ts_tablet_manager.cc:1132] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:28.791815 7919 tablet_copy_service.cc:325] P 81d8bacc3ca64d628a255ad416a77c3d: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-605a58ce874c4170b0d24d922f017068 received from {username='slave'} at 127.5.70.8:52957 I1121 10:21:28.791847 7919 tablet_copy_service.cc:413] P 81d8bacc3ca64d628a255ad416a77c3d: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-605a58ce874c4170b0d24d922f017068 on tablet 605a58ce874c4170b0d24d922f017068 with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:28.794641 9654 raft_consensus.cc:1123] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.256->[1.257-1.257] Dedup: 1.257->[] I1121 10:21:28.818720 9654 raft_consensus.cc:1123] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.300->[1.301-1.301] Dedup: 1.301->[] I1121 10:21:28.906224 10268 ts_tablet_manager.cc:721] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: Initiating tablet copy from peer 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445) I1121 10:21:28.906275 10268 tablet_copy_client.cc:243] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.5:36445 I1121 10:21:28.906481 7919 tablet_copy_service.cc:135] P 81d8bacc3ca64d628a255ad416a77c3d: Received BeginTabletCopySession request for tablet 7cd3c7862a7c420cb0b87bd87a770d84 from peer 51f14034c89344f79f40de55c5e1cfc9 ({username='slave'} at 127.5.70.8:52957) I1121 10:21:28.906514 7919 tablet_copy_service.cc:156] P 81d8bacc3ca64d628a255ad416a77c3d: Beginning new tablet copy session on tablet 7cd3c7862a7c420cb0b87bd87a770d84 from peer 51f14034c89344f79f40de55c5e1cfc9 at {username='slave'} at 127.5.70.8:52957: session id = 51f14034c89344f79f40de55c5e1cfc9-7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:28.906978 7919 tablet_copy_source_session.cc:204] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:28.907383 10268 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7cd3c7862a7c420cb0b87bd87a770d84. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:28.908569 10268 tablet_copy_client.cc:581] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 0 data blocks... I1121 10:21:28.908695 10268 tablet_copy_client.cc:544] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Starting download of 1 WAL segments... I1121 10:21:28.909932 10268 tablet_copy_client.cc:420] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:28.910521 10268 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap starting. I1121 10:21:28.930606 10268 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=296 overwritten=0 applied=296 ignored=0} inserts{seen=987 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:28.930825 10268 tablet_bootstrap.cc:439] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: Bootstrap complete. I1121 10:21:28.930971 10268 ts_tablet_manager.cc:1104] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.002s I1121 10:21:28.931145 10268 raft_consensus.cc:337] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.931167 10268 raft_consensus.cc:702] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 51f14034c89344f79f40de55c5e1cfc9, State: Initialized, Role: LEARNER I1121 10:21:28.931221 10268 consensus_queue.cc:231] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 296, Last appended: 1.296, Last appended by leader: 296, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: true } } I1121 10:21:28.931309 10268 ts_tablet_manager.cc:1132] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:28.931597 7919 tablet_copy_service.cc:325] P 81d8bacc3ca64d628a255ad416a77c3d: Request end of tablet copy session 51f14034c89344f79f40de55c5e1cfc9-7cd3c7862a7c420cb0b87bd87a770d84 received from {username='slave'} at 127.5.70.8:52957 I1121 10:21:28.931632 7919 tablet_copy_service.cc:413] P 81d8bacc3ca64d628a255ad416a77c3d: ending tablet copy session 51f14034c89344f79f40de55c5e1cfc9-7cd3c7862a7c420cb0b87bd87a770d84 on tablet 7cd3c7862a7c420cb0b87bd87a770d84 with peer 51f14034c89344f79f40de55c5e1cfc9 I1121 10:21:28.995796 9654 raft_consensus.cc:1123] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.239->[1.240-1.240] Dedup: 1.240->[] I1121 10:21:29.211921 10359 raft_consensus.cc:971] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:29.212196 10359 consensus_queue.cc:208] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.301, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:29.212874 7893 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch) I1121 10:21:29.212863 9654 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch) I1121 10:21:29.212904 8021 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch) I1121 10:21:29.213064 8660 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch) I1121 10:21:29.213125 10415 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s I1121 10:21:29.213186 10115 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s I1121 10:21:29.213244 10359 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s I1121 10:21:29.213304 10414 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s I1121 10:21:29.213301 7648 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch) I1121 10:21:29.213501 10415 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s I1121 10:21:29.214404 10115 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.302: config changed from index 301 to 302, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.214910 7893 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.214956 7648 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.215641 9654 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.215835 8021 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.216188 7271 catalog_manager.cc:4143] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d reported cstate change: config changed from index 301 to 302, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.216990 8660 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 301 to 302, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false replace: true } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.218454 7402 consensus_queue.cc:208] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:29.218868 7648 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch) I1121 10:21:29.218910 8021 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch) I1121 10:21:29.219102 9654 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch) I1121 10:21:29.219105 10414 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s I1121 10:21:29.219164 7893 raft_consensus.cc:1181] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch) I1121 10:21:29.219306 10414 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s I1121 10:21:29.219409 10415 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s I1121 10:21:29.219524 10359 consensus_queue.cc:984] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s I1121 10:21:29.220358 10360 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.220440 9654 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.221340 7262 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 5a9a3a26473340c381d502b8e9e1660c with cas_config_opid_index 302: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:29.221361 8021 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.221401 7276 catalog_manager.cc:4143] T 5a9a3a26473340c381d502b8e9e1660c P 51f14034c89344f79f40de55c5e1cfc9 reported cstate change: config changed from index 302 to 303, VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) evicted. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.221505 7648 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.222612 7893 raft_consensus.cc:2801] T 5a9a3a26473340c381d502b8e9e1660c P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, VOTER 522e51b66e2c425091349ca2c9059244 (127.5.70.7) evicted. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.225349 8641 tablet_service.cc:1067] Processing DeleteTablet for tablet 5a9a3a26473340c381d502b8e9e1660c with delete_type TABLET_DATA_TOMBSTONED (TS 522e51b66e2c425091349ca2c9059244 not found in new config with opid_index 303) from {username='slave'} at 127.0.0.1:54034 I1121 10:21:29.225512 10428 tablet_replica.cc:282] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: stopping tablet replica I1121 10:21:29.225590 10428 raft_consensus.cc:2156] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:29.225632 10428 raft_consensus.cc:2183] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:29.226526 10428 ts_tablet_manager.cc:1457] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:29.228129 10428 ts_tablet_manager.cc:1470] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.302 I1121 10:21:29.228173 10428 log.cc:1153] T 5a9a3a26473340c381d502b8e9e1660c P 522e51b66e2c425091349ca2c9059244: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal/wals/5a9a3a26473340c381d502b8e9e1660c I1121 10:21:29.228715 7263 catalog_manager.cc:3512] TS 522e51b66e2c425091349ca2c9059244 (127.5.70.7:38147): tablet 5a9a3a26473340c381d502b8e9e1660c (table rebalance_test_table_2 [id=a882fd9fc4ab493f9a0c3cc9647366e6]) successfully deleted I1121 10:21:29.375033 10414 raft_consensus.cc:971] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:29.375277 10414 consensus_queue.cc:208] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:29.375897 9654 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch) I1121 10:21:29.375895 7772 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch) I1121 10:21:29.375947 8021 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch) I1121 10:21:29.375982 8660 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch) I1121 10:21:29.376076 7893 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch) I1121 10:21:29.376189 10360 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s I1121 10:21:29.376282 10115 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s I1121 10:21:29.376333 10359 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s I1121 10:21:29.376410 10360 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s I1121 10:21:29.376505 10475 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s I1121 10:21:29.377450 10359 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.377599 9654 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.377630 7893 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.378912 7275 catalog_manager.cc:4143] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 257 to 258, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:29.378974 8660 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.379022 8021 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.379945 7772 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 257 to 258, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.380918 7402 consensus_queue.cc:208] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 258, Committed index: 258, Last appended: 1.258, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:29.381284 8660 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch) I1121 10:21:29.381322 9654 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch) I1121 10:21:29.381352 7772 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch) I1121 10:21:29.381469 8021 raft_consensus.cc:1181] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 63b9dc8cf49b47ebb644515af2f427e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 258. Preceding OpId from leader: term: 1 index: 259. (index mismatch) I1121 10:21:29.381525 10414 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s I1121 10:21:29.381590 10415 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s I1121 10:21:29.381657 10475 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s I1121 10:21:29.381788 10359 consensus_queue.cc:984] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 259, Last known committed idx: 258, Time since last communication: 0.000s I1121 10:21:29.382289 9650 raft_consensus.cc:1123] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.281->[1.282-1.282] Dedup: 1.282->[] I1121 10:21:29.382527 10415 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 LEADER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.382885 8660 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.383330 7262 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 7d59736e513f426095bedf84bbeec48d with cas_config_opid_index 258: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:29.383630 7276 catalog_manager.cc:4143] T 7d59736e513f426095bedf84bbeec48d P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 258 to 259, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New cstate: current_term: 1 leader_uuid: "63b9dc8cf49b47ebb644515af2f427e7" committed_config { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:29.383631 7772 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.383683 9654 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.384457 7874 tablet_service.cc:1067] Processing DeleteTablet for tablet 7d59736e513f426095bedf84bbeec48d with delete_type TABLET_DATA_TOMBSTONED (TS 81d8bacc3ca64d628a255ad416a77c3d not found in new config with opid_index 259) from {username='slave'} at 127.0.0.1:37730 I1121 10:21:29.384837 8021 raft_consensus.cc:2801] T 7d59736e513f426095bedf84bbeec48d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.259: config changed from index 258 to 259, VOTER 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5) evicted. New config: { opid_index: 259 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.385069 10481 tablet_replica.cc:282] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d: stopping tablet replica I1121 10:21:29.385119 10481 raft_consensus.cc:2156] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:29.385141 10481 raft_consensus.cc:2183] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:29.386068 10481 ts_tablet_manager.cc:1457] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:29.387470 10481 ts_tablet_manager.cc:1470] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.258 I1121 10:21:29.387509 10481 log.cc:1153] T 7d59736e513f426095bedf84bbeec48d P 81d8bacc3ca64d628a255ad416a77c3d: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal/wals/7d59736e513f426095bedf84bbeec48d I1121 10:21:29.387869 7263 catalog_manager.cc:3512] TS 81d8bacc3ca64d628a255ad416a77c3d (127.5.70.5:36445): tablet 7d59736e513f426095bedf84bbeec48d (table rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]) successfully deleted I1121 10:21:29.452541 10387 raft_consensus.cc:971] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:29.452806 10387 consensus_queue.cc:208] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 1.240, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:29.453305 7525 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch) I1121 10:21:29.453441 8021 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch) I1121 10:21:29.453459 7402 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch) I1121 10:21:29.453538 9654 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch) I1121 10:21:29.453635 10484 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s I1121 10:21:29.453749 10387 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s I1121 10:21:29.453758 7648 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch) I1121 10:21:29.453971 10483 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s I1121 10:21:29.454089 10387 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s I1121 10:21:29.454206 9967 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s I1121 10:21:29.454910 10483 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.241: config changed from index 240 to 241, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.455042 9654 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 240 to 241, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.455193 7648 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 240 to 241, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.455751 7525 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 240 to 241, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.456216 8021 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 240 to 241, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.456696 7276 catalog_manager.cc:4143] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 reported cstate change: config changed from index 240 to 241, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.457710 7402 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 240 to 241, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { replace: true } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.458034 7893 consensus_queue.cc:208] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 241, Committed index: 241, Last appended: 1.241, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:29.458580 7648 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch) I1121 10:21:29.458770 9654 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch) I1121 10:21:29.458770 7525 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch) I1121 10:21:29.458813 10387 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s I1121 10:21:29.459048 8021 raft_consensus.cc:1181] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 241. Preceding OpId from leader: term: 1 index: 242. (index mismatch) I1121 10:21:29.459050 9967 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s I1121 10:21:29.459143 9967 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s I1121 10:21:29.459214 9967 consensus_queue.cc:984] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 242, Last known committed idx: 241, Time since last communication: 0.000s I1121 10:21:29.459987 9967 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.242: config changed from index 241 to 242, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.460098 8021 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index 241 to 242, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.460170 9654 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index 241 to 242, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.460170 7525 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P 41118f5c2e8f48179ffbebbfb6da2edf [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index 241 to 242, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.461216 7278 catalog_manager.cc:4143] T c5abf7a6c891418482e200e7115e2c2d P 51f14034c89344f79f40de55c5e1cfc9 reported cstate change: config changed from index 241 to 242, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.461241 7648 raft_consensus.cc:2801] T c5abf7a6c891418482e200e7115e2c2d P f5bc6d2f9c574b4db9396f9f900c1141 [term 1 FOLLOWER]: Committing config change with OpId 1.242: config changed from index 241 to 242, VOTER 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1) evicted. New config: { opid_index: 242 OBSOLETE_local: false peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "f5bc6d2f9c574b4db9396f9f900c1141" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 43909 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "41118f5c2e8f48179ffbebbfb6da2edf" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.462285 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet c5abf7a6c891418482e200e7115e2c2d with cas_config_opid_index 241: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:29.462293 7370 tablet_service.cc:1067] Processing DeleteTablet for tablet c5abf7a6c891418482e200e7115e2c2d with delete_type TABLET_DATA_TOMBSTONED (TS 63b9dc8cf49b47ebb644515af2f427e7 not found in new config with opid_index 242) from {username='slave'} at 127.0.0.1:34708 I1121 10:21:29.462815 10490 tablet_replica.cc:282] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7: stopping tablet replica I1121 10:21:29.462864 10490 raft_consensus.cc:2156] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:29.462883 10490 raft_consensus.cc:2183] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:29.463722 10490 ts_tablet_manager.cc:1457] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:29.465248 10490 ts_tablet_manager.cc:1470] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.241 I1121 10:21:29.465288 10490 log.cc:1153] T c5abf7a6c891418482e200e7115e2c2d P 63b9dc8cf49b47ebb644515af2f427e7: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/c5abf7a6c891418482e200e7115e2c2d I1121 10:21:29.465641 7262 catalog_manager.cc:3512] TS 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995): tablet c5abf7a6c891418482e200e7115e2c2d (table rebalance_test_table_1 [id=1b28b579b64d471781f82b9f40e804ca]) successfully deleted I1121 10:21:29.517791 9654 raft_consensus.cc:1123] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.295->[1.296-1.296] Dedup: 1.296->[] I1121 10:21:29.836544 10483 raft_consensus.cc:971] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:29.836774 10483 consensus_queue.cc:208] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 1.282, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:29.837473 7402 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch) I1121 10:21:29.837481 8021 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch) I1121 10:21:29.837486 8660 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch) I1121 10:21:29.837488 7772 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch) I1121 10:21:29.837663 9654 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 282. Preceding OpId from leader: term: 1 index: 283. (index mismatch) I1121 10:21:29.837764 10483 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s I1121 10:21:29.837846 9967 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s I1121 10:21:29.837919 10387 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s I1121 10:21:29.838008 10484 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s I1121 10:21:29.838146 10532 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s I1121 10:21:29.839094 10483 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.283: config changed from index 282 to 283, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.839295 8660 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.839318 9654 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.839603 7772 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.840857 7273 catalog_manager.cc:4143] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 282 to 283, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.841907 8021 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.841931 7402 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.283: config changed from index 282 to 283, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.843544 7893 consensus_queue.cc:208] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 1.283, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:29.843984 7402 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch) I1121 10:21:29.844018 7772 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch) I1121 10:21:29.844146 9654 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch) I1121 10:21:29.844178 8660 raft_consensus.cc:1181] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch) I1121 10:21:29.844202 10387 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s I1121 10:21:29.844278 10387 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s I1121 10:21:29.844353 10387 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s I1121 10:21:29.844419 10483 consensus_queue.cc:984] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s I1121 10:21:29.845176 10539 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.284: config changed from index 283 to 284, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.845331 7402 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 283 to 284, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.845505 8660 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 283 to 284, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.846369 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 605a58ce874c4170b0d24d922f017068 with cas_config_opid_index 283: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:29.846493 7273 catalog_manager.cc:4143] T 605a58ce874c4170b0d24d922f017068 P 63b9dc8cf49b47ebb644515af2f427e7 reported cstate change: config changed from index 283 to 284, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.846666 9654 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 283 to 284, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.846915 7772 raft_consensus.cc:2801] T 605a58ce874c4170b0d24d922f017068 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 283 to 284, VOTER c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6) evicted. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } } peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:29.847183 8003 tablet_service.cc:1067] Processing DeleteTablet for tablet 605a58ce874c4170b0d24d922f017068 with delete_type TABLET_DATA_TOMBSTONED (TS c9e9c35ff1cb4b68ba7cf5c037a98347 not found in new config with opid_index 284) from {username='slave'} at 127.0.0.1:44220 I1121 10:21:29.847941 10541 tablet_replica.cc:282] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347: stopping tablet replica I1121 10:21:29.848044 10541 raft_consensus.cc:2156] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:29.848117 10541 raft_consensus.cc:2183] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:29.849335 10541 ts_tablet_manager.cc:1457] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:29.850908 10541 ts_tablet_manager.cc:1470] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.283 I1121 10:21:29.850945 10541 log.cc:1153] T 605a58ce874c4170b0d24d922f017068 P c9e9c35ff1cb4b68ba7cf5c037a98347: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal/wals/605a58ce874c4170b0d24d922f017068 I1121 10:21:29.851420 7262 catalog_manager.cc:3512] TS c9e9c35ff1cb4b68ba7cf5c037a98347 (127.5.70.6:32833): tablet 605a58ce874c4170b0d24d922f017068 (table rebalance_test_table_0 [id=5bf580d0415a45fc82dac002a52941f4]) successfully deleted I1121 10:21:30.012228 10532 raft_consensus.cc:971] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d: attempting to promote NON_VOTER 51f14034c89344f79f40de55c5e1cfc9 to VOTER I1121 10:21:30.012480 10532 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.296, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:30.013020 7772 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch) I1121 10:21:30.013036 7402 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch) I1121 10:21:30.013059 8660 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch) I1121 10:21:30.013089 8021 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch) I1121 10:21:30.013278 10540 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s I1121 10:21:30.013334 9654 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 LEARNER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch) I1121 10:21:30.013356 10539 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s I1121 10:21:30.013447 10484 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s I1121 10:21:30.013530 10484 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s I1121 10:21:30.013622 10483 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s I1121 10:21:30.014554 10540 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.297: config changed from index 296 to 297, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.014684 7772 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 296 to 297, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.014843 8021 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 296 to 297, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.015916 7402 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 296 to 297, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.016055 7278 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 reported cstate change: config changed from index 296 to 297, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.016407 8660 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 296 to 297, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.017443 7893 consensus_queue.cc:208] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 1.297, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } I1121 10:21:30.017604 9654 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 296 to 297, 51f14034c89344f79f40de55c5e1cfc9 (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "1b3038ad6c564f438aa18383cddc769a" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 32993 } attrs { replace: true } } peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.018193 8021 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch) I1121 10:21:30.018188 7402 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch) I1121 10:21:30.018426 9650 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch) I1121 10:21:30.018481 9967 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s I1121 10:21:30.018550 10387 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s I1121 10:21:30.018714 8660 raft_consensus.cc:1181] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Refusing update from remote peer 81d8bacc3ca64d628a255ad416a77c3d: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch) I1121 10:21:30.018883 10387 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s I1121 10:21:30.018957 10387 consensus_queue.cc:984] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 297, Time since last communication: 0.000s I1121 10:21:30.019608 10387 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 81d8bacc3ca64d628a255ad416a77c3d [term 1 LEADER]: Committing config change with OpId 1.298: config changed from index 297 to 298, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.019771 9650 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 51f14034c89344f79f40de55c5e1cfc9 [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 297 to 298, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.019773 8660 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 297 to 298, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.019778 8021 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P c9e9c35ff1cb4b68ba7cf5c037a98347 [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 297 to 298, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.020774 7275 catalog_manager.cc:4143] T 7cd3c7862a7c420cb0b87bd87a770d84 P 522e51b66e2c425091349ca2c9059244 reported cstate change: config changed from index 297 to 298, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "81d8bacc3ca64d628a255ad416a77c3d" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.021039 7263 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 7cd3c7862a7c420cb0b87bd87a770d84 with cas_config_opid_index 297: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:30.021131 7402 raft_consensus.cc:2801] T 7cd3c7862a7c420cb0b87bd87a770d84 P 63b9dc8cf49b47ebb644515af2f427e7 [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 297 to 298, VOTER 1b3038ad6c564f438aa18383cddc769a (127.5.70.4) evicted. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "81d8bacc3ca64d628a255ad416a77c3d" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 36445 } } peers { permanent_uuid: "c9e9c35ff1cb4b68ba7cf5c037a98347" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 32833 } } peers { permanent_uuid: "522e51b66e2c425091349ca2c9059244" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 38147 } attrs { promote: false } } peers { permanent_uuid: "63b9dc8cf49b47ebb644515af2f427e7" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 36995 } attrs { promote: false } } peers { permanent_uuid: "51f14034c89344f79f40de55c5e1cfc9" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 44901 } attrs { promote: false } } } I1121 10:21:30.021966 7754 tablet_service.cc:1067] Processing DeleteTablet for tablet 7cd3c7862a7c420cb0b87bd87a770d84 with delete_type TABLET_DATA_TOMBSTONED (TS 1b3038ad6c564f438aa18383cddc769a not found in new config with opid_index 298) from {username='slave'} at 127.0.0.1:42904 I1121 10:21:30.022217 10563 tablet_replica.cc:282] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a: stopping tablet replica I1121 10:21:30.022270 10563 raft_consensus.cc:2156] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:30.022287 10563 raft_consensus.cc:2183] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:30.023340 10563 ts_tablet_manager.cc:1457] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:30.024840 10563 ts_tablet_manager.cc:1470] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.297 I1121 10:21:30.024893 10563 log.cc:1153] T 7cd3c7862a7c420cb0b87bd87a770d84 P 1b3038ad6c564f438aa18383cddc769a: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:30.025353 7263 catalog_manager.cc:3512] TS 1b3038ad6c564f438aa18383cddc769a (127.5.70.4:32993): tablet 7cd3c7862a7c420cb0b87bd87a770d84 (table rebalance_test_table_3 [id=a6818c1427a0476183a4cee5788608f5]) successfully deleted Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- 913850c84b1540c1945d797659799e2b | 127.5.70.62:38839 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 1b3038ad6c564f438aa18383cddc769a | 127.5.70.4:32993 | HEALTHY | 41118f5c2e8f48179ffbebbfb6da2edf | 127.5.70.2:39291 | HEALTHY | 51f14034c89344f79f40de55c5e1cfc9 | 127.5.70.8:44901 | HEALTHY | 522e51b66e2c425091349ca2c9059244 | 127.5.70.7:38147 | HEALTHY | 63b9dc8cf49b47ebb644515af2f427e7 | 127.5.70.1:36995 | HEALTHY | 81d8bacc3ca64d628a255ad416a77c3d | 127.5.70.5:36445 | HEALTHY | c9e9c35ff1cb4b68ba7cf5c037a98347 | 127.5.70.6:32833 | HEALTHY | f5bc6d2f9c574b4db9396f9f900c1141 | 127.5.70.3:43909 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 8 Flag | Value | Tags | Tablet Server ------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 8 server(s) checked local_ip_for_outbound_sockets | 127.5.70.1 | experimental | 127.5.70.1:36995 local_ip_for_outbound_sockets | 127.5.70.2 | experimental | 127.5.70.2:39291 local_ip_for_outbound_sockets | 127.5.70.3 | experimental | 127.5.70.3:43909 local_ip_for_outbound_sockets | 127.5.70.4 | experimental | 127.5.70.4:32993 local_ip_for_outbound_sockets | 127.5.70.5 | experimental | 127.5.70.5:36445 local_ip_for_outbound_sockets | 127.5.70.6 | experimental | 127.5.70.6:32833 local_ip_for_outbound_sockets | 127.5.70.7 | experimental | 127.5.70.7:38147 local_ip_for_outbound_sockets | 127.5.70.8 | experimental | 127.5.70.8:44901 never_fsync | true | unsafe,advanced | all 8 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked server_dump_info_format | pb | hidden | all 8 server(s) checked server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.70.1:36995 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.70.2:39291 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.70.3:43909 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.70.4:32993 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.70.5:36445 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.70.6:32833 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.70.7:38147 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_7.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.5.70.8:44901 Version Summary Version | Servers -----------------+------------------------- 1.12.0-SNAPSHOT | all 9 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ------------------------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0 rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 10 First Quartile | 10 Median | 10 Third Quartile | 10 Maximum | 10 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 8 Tables | 6 Tablets | 16 Replicas | 80 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1121 10:21:30.243896 5400 log_verifier.cc:121] Checking tablet 133fea61ffe5433785a3fe3736a2a668 I1121 10:21:30.282184 5400 log_verifier.cc:172] Verified matching terms for 273 ops in tablet 133fea61ffe5433785a3fe3736a2a668 I1121 10:21:30.282245 5400 log_verifier.cc:121] Checking tablet 169c4cf587a949748d1734c65258b7a7 I1121 10:21:30.326319 5400 log_verifier.cc:172] Verified matching terms for 344 ops in tablet 169c4cf587a949748d1734c65258b7a7 I1121 10:21:30.326393 5400 log_verifier.cc:121] Checking tablet 55d18b1e7ede4cb28f5dfc4ded166a30 I1121 10:21:30.361076 5400 log_verifier.cc:172] Verified matching terms for 252 ops in tablet 55d18b1e7ede4cb28f5dfc4ded166a30 I1121 10:21:30.361130 5400 log_verifier.cc:121] Checking tablet 5a9a3a26473340c381d502b8e9e1660c I1121 10:21:30.401681 5400 log_verifier.cc:172] Verified matching terms for 303 ops in tablet 5a9a3a26473340c381d502b8e9e1660c I1121 10:21:30.401842 5400 log_verifier.cc:121] Checking tablet 5c8cdc07bf6f4480b269ca34a9b3e426 I1121 10:21:30.440102 5400 log_verifier.cc:172] Verified matching terms for 273 ops in tablet 5c8cdc07bf6f4480b269ca34a9b3e426 I1121 10:21:30.440158 5400 log_verifier.cc:121] Checking tablet 605a58ce874c4170b0d24d922f017068 I1121 10:21:30.482856 5400 log_verifier.cc:172] Verified matching terms for 284 ops in tablet 605a58ce874c4170b0d24d922f017068 I1121 10:21:30.482918 5400 log_verifier.cc:121] Checking tablet 787130b10eda4a71b2212bdb7ed2f140 I1121 10:21:30.515743 5400 log_verifier.cc:172] Verified matching terms for 240 ops in tablet 787130b10eda4a71b2212bdb7ed2f140 I1121 10:21:30.515795 5400 log_verifier.cc:121] Checking tablet 7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:30.556375 5400 log_verifier.cc:172] Verified matching terms for 298 ops in tablet 7cd3c7862a7c420cb0b87bd87a770d84 I1121 10:21:30.556437 5400 log_verifier.cc:121] Checking tablet 7d59736e513f426095bedf84bbeec48d I1121 10:21:30.594795 5400 log_verifier.cc:172] Verified matching terms for 259 ops in tablet 7d59736e513f426095bedf84bbeec48d I1121 10:21:30.594868 5400 log_verifier.cc:121] Checking tablet 865e88694908474197879cd8d34789ad I1121 10:21:30.637962 5400 log_verifier.cc:172] Verified matching terms for 361 ops in tablet 865e88694908474197879cd8d34789ad I1121 10:21:30.638022 5400 log_verifier.cc:121] Checking tablet 993b9cf62f6c4e6296acbb32382ab037 I1121 10:21:30.678508 5400 log_verifier.cc:172] Verified matching terms for 291 ops in tablet 993b9cf62f6c4e6296acbb32382ab037 I1121 10:21:30.678566 5400 log_verifier.cc:121] Checking tablet b0cf374217f64043803515e5125b6de0 I1121 10:21:30.693282 5400 log_verifier.cc:172] Verified matching terms for 4 ops in tablet b0cf374217f64043803515e5125b6de0 I1121 10:21:30.693315 5400 log_verifier.cc:121] Checking tablet c2bd898e6d3340f5982c1386e0ebd41a I1121 10:21:30.732374 5400 log_verifier.cc:172] Verified matching terms for 323 ops in tablet c2bd898e6d3340f5982c1386e0ebd41a I1121 10:21:30.732434 5400 log_verifier.cc:121] Checking tablet c5abf7a6c891418482e200e7115e2c2d I1121 10:21:30.768124 5400 log_verifier.cc:172] Verified matching terms for 242 ops in tablet c5abf7a6c891418482e200e7115e2c2d I1121 10:21:30.768177 5400 log_verifier.cc:121] Checking tablet d84faff8cd0f433d8de8664fd0054d2d I1121 10:21:30.806970 5400 log_verifier.cc:172] Verified matching terms for 292 ops in tablet d84faff8cd0f433d8de8664fd0054d2d I1121 10:21:30.807031 5400 log_verifier.cc:121] Checking tablet f30bc779dfa840c6ae504b94d82eaa5a I1121 10:21:30.844295 5400 log_verifier.cc:172] Verified matching terms for 253 ops in tablet f30bc779dfa840c6ae504b94d82eaa5a I1121 10:21:30.845003 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 7323 W1121 10:21:30.850334 7459 connection.cc:655] client connection to 127.5.70.1:36995 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W1121 10:21:30.850417 7459 consensus_peers.cc:473] T c2bd898e6d3340f5982c1386e0ebd41a P 41118f5c2e8f48179ffbebbfb6da2edf -> Peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995): Couldn't send request to peer 63b9dc8cf49b47ebb644515af2f427e7. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. W1121 10:21:30.850735 7831 connection.cc:655] client connection to 127.5.70.1:36995 recv error: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107) W1121 10:21:30.850791 7831 consensus_peers.cc:473] T 605a58ce874c4170b0d24d922f017068 P 81d8bacc3ca64d628a255ad416a77c3d -> Peer 63b9dc8cf49b47ebb644515af2f427e7 (127.5.70.1:36995): Couldn't send request to peer 63b9dc8cf49b47ebb644515af2f427e7. Status: Network error: recv error from 0.0.0.0:0: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry. I1121 10:21:30.851071 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 7449 I1121 10:21:30.855521 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 7573 I1121 10:21:30.860054 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 7697 I1121 10:21:30.863673 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 7821 I1121 10:21:30.867801 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 7945 I1121 10:21:30.871230 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 8582 I1121 10:21:30.875164 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 9576 I1121 10:21:30.879298 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master with pid 7252 2019-11-21T10:21:30Z chronyd exiting [ OK ] RebalanceParamTest.Rebalance/7 (17532 ms) [----------] 1 test from RebalanceParamTest (17532 ms total) [----------] 1 test from TserverAddedDuringRebalancingTest [ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1 I1121 10:21:30.956887 5400 test_util.cc:225] Using random seed: -1456293806 I1121 10:21:30.957262 5400 ts_itest-base.cc:112] Starting cluster with: I1121 10:21:30.957276 5400 ts_itest-base.cc:113] -------------- I1121 10:21:30.957279 5400 ts_itest-base.cc:114] 8 tablet servers I1121 10:21:30.957281 5400 ts_itest-base.cc:115] 3 replicas per TS I1121 10:21:30.957284 5400 ts_itest-base.cc:116] -------------- 2019-11-21T10:21:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2019-11-21T10:21:30Z Disabled control of system clock I1121 10:21:30.964824 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:45301 --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tserver_unresponsive_timeout_ms=3000 --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:31.008689 10682 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:31.008858 10682 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:31.009748 10682 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:31.009770 10682 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W1121 10:21:31.009775 10682 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:31.009778 10682 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W1121 10:21:31.009783 10682 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I1121 10:21:31.011101 10682 master_main.cc:98] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --tserver_unresponsive_timeout_ms=3000 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:45301 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.10682 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:31.012926 10682 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:31.014065 10682 hybrid_clock.cc:183] HybridClock initialized: now 1574331691014041 us; error 32 us; skew 500 ppm I1121 10:21:31.014241 10682 fs_manager.cc:261] Metadata directory not provided I1121 10:21:31.014276 10682 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:31.014405 10682 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:31.014431 10682 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:31.015506 10682 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/instance: uuid: "c7a94ae451594629a208a04c2a14aa18" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.015704 10682 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "c7a94ae451594629a208a04c2a14aa18" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.017208 10682 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:21:31.017580 10682 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:31.017722 10682 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:31.018448 10682 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:31.018491 10682 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal uuid: "c7a94ae451594629a208a04c2a14aa18" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.018548 10682 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:31.040652 10682 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:31.043445 10682 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.62:45301 I1121 10:21:31.043762 10682 webserver.cc:382] Webserver started at http://127.5.70.62:41209/ using document root and password file I1121 10:21:31.044011 10682 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb I1121 10:21:31.044632 10743 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:31.046931 10743 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18: Bootstrap starting. I1121 10:21:31.047380 10743 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18: No blocks or log segments found. Creating new log. I1121 10:21:31.047582 10743 log.cc:810] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18: Log is configured to *not* fsync() on all Append() calls I1121 10:21:31.048081 10743 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18: No bootstrap required, opened a new log I1121 10:21:31.048110 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master as pid 10682 I1121 10:21:31.048557 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.70.1:0 --local_ip_for_outbound_sockets=127.5.70.1 --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} I1121 10:21:31.050081 10743 raft_consensus.cc:337] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a94ae451594629a208a04c2a14aa18" member_type: VOTER } I1121 10:21:31.050153 10743 raft_consensus.cc:363] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:31.050174 10743 raft_consensus.cc:702] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a94ae451594629a208a04c2a14aa18, State: Initialized, Role: FOLLOWER I1121 10:21:31.050334 10743 consensus_queue.cc:231] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a94ae451594629a208a04c2a14aa18" member_type: VOTER } I1121 10:21:31.050365 10743 raft_consensus.cc:377] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:21:31.050379 10743 raft_consensus.cc:469] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:21:31.050396 10743 raft_consensus.cc:2905] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:31.050827 10743 raft_consensus.cc:491] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a94ae451594629a208a04c2a14aa18" member_type: VOTER } I1121 10:21:31.050894 10743 leader_election.cc:310] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7a94ae451594629a208a04c2a14aa18; no voters: I1121 10:21:31.050982 10743 leader_election.cc:296] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:21:31.051080 10743 sys_catalog.cc:340] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a94ae451594629a208a04c2a14aa18" member_type: VOTER } } I1121 10:21:31.051105 10743 sys_catalog.cc:343] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [sys.catalog]: This master's current role is: FOLLOWER I1121 10:21:31.051182 10743 sys_catalog.cc:428] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [sys.catalog]: configured and running, proceeding with master startup. I1121 10:21:31.052050 10746 raft_consensus.cc:2653] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:31.052161 10746 raft_consensus.cc:664] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [term 1 LEADER]: Becoming Leader. State: Replica: c7a94ae451594629a208a04c2a14aa18, State: Running, Role: LEADER I1121 10:21:31.052242 10746 consensus_queue.cc:208] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a94ae451594629a208a04c2a14aa18" member_type: VOTER } I1121 10:21:31.052439 10746 sys_catalog.cc:340] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7a94ae451594629a208a04c2a14aa18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a94ae451594629a208a04c2a14aa18" member_type: VOTER } } I1121 10:21:31.052459 10746 sys_catalog.cc:343] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [sys.catalog]: This master's current role is: LEADER I1121 10:21:31.052593 10747 sys_catalog.cc:340] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7a94ae451594629a208a04c2a14aa18. Latest consensus state: current_term: 1 leader_uuid: "c7a94ae451594629a208a04c2a14aa18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a94ae451594629a208a04c2a14aa18" member_type: VOTER } } I1121 10:21:31.052613 10747 sys_catalog.cc:343] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18 [sys.catalog]: This master's current role is: LEADER W1121 10:21:31.052950 10748 catalog_manager.cc:1159] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18: acquiring CA information for follower catalog manager: Not found: root CA entry not found W1121 10:21:31.052971 10748 catalog_manager.cc:631] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I1121 10:21:31.053594 10751 catalog_manager.cc:1090] Loading table and tablet metadata into memory... I1121 10:21:31.053741 10751 catalog_manager.cc:1101] Initializing Kudu internal certificate authority... I1121 10:21:31.068820 10751 catalog_manager.cc:985] Generated new certificate authority record I1121 10:21:31.069443 10751 catalog_manager.cc:1110] Loading token signing keys... I1121 10:21:31.077623 10751 catalog_manager.cc:4455] T 00000000000000000000000000000000 P c7a94ae451594629a208a04c2a14aa18: Generated new TSK 0 I1121 10:21:31.077738 10751 catalog_manager.cc:1120] Initializing in-progress tserver states... WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:31.103011 10745 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:31.103235 10745 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:31.104383 10745 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:31.104411 10745 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:31.104432 10745 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.1 I1121 10:21:31.106127 10745 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.70.1 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.10745 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:31.108839 10745 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:31.110003 10745 hybrid_clock.cc:183] HybridClock initialized: now 1574331691109983 us; error 30 us; skew 500 ppm I1121 10:21:31.110188 10745 fs_manager.cc:261] Metadata directory not provided I1121 10:21:31.110249 10745 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:31.110379 10745 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:31.110409 10745 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:31.111898 10745 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "cd693d36f0014dc3a7c3f26196124794" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.112161 10745 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "cd693d36f0014dc3a7c3f26196124794" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.114339 10745 fs_manager.cc:508] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s I1121 10:21:31.114727 10745 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:31.114869 10745 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:31.115820 10745 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:31.115867 10745 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal uuid: "cd693d36f0014dc3a7c3f26196124794" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.115928 10745 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:31.193928 10745 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:31.194391 10745 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:31.194419 10745 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:31.199681 10745 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.1:43035 I1121 10:21:31.200093 10745 webserver.cc:382] Webserver started at http://127.5.70.1:46855/ using document root and password file I1121 10:21:31.200402 10745 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb I1121 10:21:31.205281 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 10745 I1121 10:21:31.205623 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.70.2:0 --local_ip_for_outbound_sockets=127.5.70.2 --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} I1121 10:21:31.209815 10875 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:31.209874 10875 heartbeater.cc:406] Registering TS with master... I1121 10:21:31.209976 10875 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:31.210386 10710 ts_manager.cc:192] Registered new tserver with Master: cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:31.211174 10710 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.1:41173 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:31.246902 10879 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:31.247051 10879 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:31.247859 10879 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:31.247877 10879 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:31.247892 10879 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.2 I1121 10:21:31.249089 10879 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.70.2 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.10879 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.2 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:31.250874 10879 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:31.252039 10879 hybrid_clock.cc:183] HybridClock initialized: now 1574331691252015 us; error 32 us; skew 500 ppm I1121 10:21:31.252163 10879 fs_manager.cc:261] Metadata directory not provided I1121 10:21:31.252193 10879 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:31.252271 10879 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:31.252288 10879 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:31.253355 10879 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "af0d02c642ed4eab8d971d9f05d32044" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.253538 10879 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "af0d02c642ed4eab8d971d9f05d32044" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.254900 10879 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I1121 10:21:31.255142 10879 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:31.255239 10879 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:31.255923 10879 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:31.255964 10879 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal uuid: "af0d02c642ed4eab8d971d9f05d32044" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.256001 10879 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:31.298085 10879 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:31.298444 10879 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:31.298463 10879 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:31.303831 10879 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.2:43797 I1121 10:21:31.304162 10879 webserver.cc:382] Webserver started at http://127.5.70.2:35089/ using document root and password file I1121 10:21:31.304419 10879 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb I1121 10:21:31.308545 11000 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:31.308600 11000 heartbeater.cc:406] Registering TS with master... I1121 10:21:31.308759 11000 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:31.309108 10710 ts_manager.cc:192] Registered new tserver with Master: af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:31.309639 10710 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.2:60771 I1121 10:21:31.311811 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 10879 I1121 10:21:31.312149 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.70.3:0 --local_ip_for_outbound_sockets=127.5.70.3 --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:31.349258 11003 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:31.349560 11003 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:31.350486 11003 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:31.350509 11003 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:31.350530 11003 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.3 I1121 10:21:31.351702 11003 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.70.3 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11003 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.3 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:31.353612 11003 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:31.354826 11003 hybrid_clock.cc:183] HybridClock initialized: now 1574331691354738 us; error 36 us; skew 500 ppm I1121 10:21:31.354984 11003 fs_manager.cc:261] Metadata directory not provided I1121 10:21:31.355024 11003 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:31.355129 11003 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:31.355149 11003 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:31.356292 11003 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.356503 11003 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.358083 11003 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s I1121 10:21:31.358355 11003 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:31.358484 11003 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:31.359256 11003 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:31.359298 11003 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.359369 11003 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:31.395295 11003 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:31.395750 11003 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:31.395777 11003 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:31.400746 11003 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.3:40217 I1121 10:21:31.401098 11003 webserver.cc:382] Webserver started at http://127.5.70.3:45453/ using document root and password file I1121 10:21:31.401358 11003 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb I1121 10:21:31.405544 11124 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:31.405591 11124 heartbeater.cc:406] Registering TS with master... I1121 10:21:31.405669 11124 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:31.406005 10710 ts_manager.cc:192] Registered new tserver with Master: ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:31.406288 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 11003 I1121 10:21:31.406563 10710 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.3:43747 I1121 10:21:31.406566 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.70.4:0 --local_ip_for_outbound_sockets=127.5.70.4 --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:31.443745 11127 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:31.443917 11127 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:31.444751 11127 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:31.444777 11127 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:31.444790 11127 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.4 I1121 10:21:31.445962 11127 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.70.4 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11127 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.4 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:31.447724 11127 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:31.448864 11127 hybrid_clock.cc:183] HybridClock initialized: now 1574331691448812 us; error 62 us; skew 500 ppm I1121 10:21:31.449018 11127 fs_manager.cc:261] Metadata directory not provided I1121 10:21:31.449059 11127 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:31.449156 11127 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:31.449184 11127 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:31.450255 11127 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "5457a73f9ecb40ee9889513a8412937b" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.450459 11127 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "5457a73f9ecb40ee9889513a8412937b" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.452037 11127 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:31.452311 11127 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:31.452435 11127 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:31.453191 11127 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:31.453230 11127 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal uuid: "5457a73f9ecb40ee9889513a8412937b" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.453291 11127 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:31.485975 11127 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:31.486344 11127 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:31.486366 11127 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:31.492383 11127 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.4:37207 I1121 10:21:31.492806 11127 webserver.cc:382] Webserver started at http://127.5.70.4:34415/ using document root and password file I1121 10:21:31.493091 11127 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb I1121 10:21:31.497313 11248 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:31.497359 11248 heartbeater.cc:406] Registering TS with master... I1121 10:21:31.497462 11248 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:31.497778 10710 ts_manager.cc:192] Registered new tserver with Master: 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:31.498328 10710 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.4:35867 I1121 10:21:31.500830 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 11127 I1121 10:21:31.501147 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.70.5:0 --local_ip_for_outbound_sockets=127.5.70.5 --webserver_interface=127.5.70.5 --webserver_port=0 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:31.538198 11251 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:31.538350 11251 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:31.539253 11251 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:31.539278 11251 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:31.539295 11251 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.5 I1121 10:21:31.540674 11251 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.5:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.70.5 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11251 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.5 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:31.542524 11251 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:31.543660 11251 hybrid_clock.cc:183] HybridClock initialized: now 1574331691543633 us; error 26 us; skew 500 ppm I1121 10:21:31.543835 11251 fs_manager.cc:261] Metadata directory not provided I1121 10:21:31.543869 11251 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:31.543977 11251 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:31.543996 11251 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:31.545097 11251 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "fed6a935d10042b48cdf1f30824f6c57" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.545312 11251 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "fed6a935d10042b48cdf1f30824f6c57" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.546715 11251 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s I1121 10:21:31.547020 11251 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:31.547150 11251 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:31.547890 11251 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:31.547928 11251 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal uuid: "fed6a935d10042b48cdf1f30824f6c57" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.547982 11251 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:31.582132 11251 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:31.582540 11251 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:31.582557 11251 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:31.588171 11251 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.5:35757 I1121 10:21:31.588507 11251 webserver.cc:382] Webserver started at http://127.5.70.5:40163/ using document root and password file I1121 10:21:31.588790 11251 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb I1121 10:21:31.592882 11372 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:31.592928 11372 heartbeater.cc:406] Registering TS with master... I1121 10:21:31.593022 11372 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:31.593343 10710 ts_manager.cc:192] Registered new tserver with Master: fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757) I1121 10:21:31.593896 10710 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.5:43661 I1121 10:21:31.595228 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 11251 I1121 10:21:31.595537 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.70.6:0 --local_ip_for_outbound_sockets=127.5.70.6 --webserver_interface=127.5.70.6 --webserver_port=0 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:31.632441 11375 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:31.632625 11375 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:31.633406 11375 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:31.633428 11375 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:31.633447 11375 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.6 I1121 10:21:31.634577 11375 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.6:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.70.6 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11375 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.6 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:31.636327 11375 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:31.637465 11375 hybrid_clock.cc:183] HybridClock initialized: now 1574331691637440 us; error 34 us; skew 500 ppm I1121 10:21:31.637616 11375 fs_manager.cc:261] Metadata directory not provided I1121 10:21:31.637656 11375 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:31.637771 11375 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:31.637795 11375 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:31.638877 11375 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "2c3332f15c014e68bed03f69d8c6287d" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.639092 11375 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "2c3332f15c014e68bed03f69d8c6287d" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.640466 11375 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:31.640786 11375 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:31.640890 11375 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:31.641516 11375 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:31.641556 11375 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal uuid: "2c3332f15c014e68bed03f69d8c6287d" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.641592 11375 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:31.682482 11375 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:31.682869 11375 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:31.682888 11375 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:31.687507 11375 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.6:37787 I1121 10:21:31.687863 11375 webserver.cc:382] Webserver started at http://127.5.70.6:42409/ using document root and password file I1121 10:21:31.688144 11375 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb I1121 10:21:31.689837 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 11375 I1121 10:21:31.690194 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.70.7:0 --local_ip_for_outbound_sockets=127.5.70.7 --webserver_interface=127.5.70.7 --webserver_port=0 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} I1121 10:21:31.692886 11496 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:31.692945 11496 heartbeater.cc:406] Registering TS with master... I1121 10:21:31.693079 11496 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:31.693450 10710 ts_manager.cc:192] Registered new tserver with Master: 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787) I1121 10:21:31.694080 10710 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.6:45045 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:31.728320 11499 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:31.728489 11499 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:31.729307 11499 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:31.729328 11499 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:31.729348 11499 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.7 I1121 10:21:31.730528 11499 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.7:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.70.7 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11499 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.7 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:31.732375 11499 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:31.733544 11499 hybrid_clock.cc:183] HybridClock initialized: now 1574331691733523 us; error 37 us; skew 500 ppm I1121 10:21:31.733681 11499 fs_manager.cc:261] Metadata directory not provided I1121 10:21:31.733757 11499 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:31.733851 11499 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:31.733870 11499 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:31.734951 11499 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "77830a7609a549e0a1c76586d0e301ce" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.735136 11499 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "77830a7609a549e0a1c76586d0e301ce" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.736574 11499 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:31.736833 11499 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:31.736932 11499 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:31.737632 11499 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:31.737673 11499 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal uuid: "77830a7609a549e0a1c76586d0e301ce" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.737711 11499 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:31.779556 11499 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:31.779937 11499 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:31.779958 11499 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:31.784476 11499 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.7:40193 I1121 10:21:31.785523 11499 webserver.cc:382] Webserver started at http://127.5.70.7:35805/ using document root and password file I1121 10:21:31.785858 11499 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb I1121 10:21:31.791327 11620 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:31.791388 11620 heartbeater.cc:406] Registering TS with master... I1121 10:21:31.791503 11620 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:31.791893 10710 ts_manager.cc:192] Registered new tserver with Master: 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193) I1121 10:21:31.792559 10710 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.7:41155 I1121 10:21:31.795068 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 11499 I1121 10:21:31.795363 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data --block_manager=log --rpc_bind_addresses=127.5.70.8:0 --local_ip_for_outbound_sockets=127.5.70.8 --webserver_interface=127.5.70.8 --webserver_port=0 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:31.832621 11623 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:31.832772 11623 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:31.833518 11623 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:31.833534 11623 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:31.833547 11623 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.8 I1121 10:21:31.834652 11623 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.8:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb --webserver_interface=127.5.70.8 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.11623 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.8 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:31.836392 11623 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:31.837523 11623 hybrid_clock.cc:183] HybridClock initialized: now 1574331691837501 us; error 38 us; skew 500 ppm I1121 10:21:31.837659 11623 fs_manager.cc:261] Metadata directory not provided I1121 10:21:31.837689 11623 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:31.837795 11623 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:31.837816 11623 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:31.838819 11623 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/instance: uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.839030 11623 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal/instance: uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.840476 11623 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:21:31.840919 11623 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:31.841053 11623 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:31.841750 11623 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:31.841790 11623 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:31.841851 11623 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:31.923645 11623 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:31.924039 11623 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:31.924058 11623 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:31.929575 11623 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.8:41471 I1121 10:21:31.929872 11623 webserver.cc:382] Webserver started at http://127.5.70.8:41913/ using document root and password file I1121 10:21:31.930150 11623 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb I1121 10:21:31.934672 11744 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:31.934778 11744 heartbeater.cc:406] Registering TS with master... I1121 10:21:31.934892 11744 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:31.935218 10710 ts_manager.cc:192] Registered new tserver with Master: 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471) I1121 10:21:31.935683 10710 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.8:39301 I1121 10:21:31.940173 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 11623 I1121 10:21:31.942874 5400 external_mini_cluster.cc:717] 8 TS(s) registered with all masters I1121 10:21:31.947413 5400 external_mini_cluster.cc:717] 8 TS(s) registered with all masters I1121 10:21:31.947451 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 11251 I1121 10:21:31.950558 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 11375 I1121 10:21:31.953550 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 11499 I1121 10:21:31.956579 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 11623 I1121 10:21:32.212076 10875 heartbeater.cc:447] Master 127.5.70.62:45301 was elected leader, sending a full tablet report... I1121 10:21:32.310367 11000 heartbeater.cc:447] Master 127.5.70.62:45301 was elected leader, sending a full tablet report... I1121 10:21:32.407378 11124 heartbeater.cc:447] Master 127.5.70.62:45301 was elected leader, sending a full tablet report... I1121 10:21:32.499001 11248 heartbeater.cc:447] Master 127.5.70.62:45301 was elected leader, sending a full tablet report... I1121 10:21:35.711663 10710 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:35.718688 11061 tablet_service.cc:1023] Processing CreateTablet for tablet 4ecac46041da45aba6e82df02354fe89 (table=rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.718899 11061 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ecac46041da45aba6e82df02354fe89. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.718941 11176 tablet_service.cc:1023] Processing CreateTablet for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 (table=rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.719143 11176 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.719545 10936 tablet_service.cc:1023] Processing CreateTablet for tablet 4ecac46041da45aba6e82df02354fe89 (table=rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.719727 10936 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ecac46041da45aba6e82df02354fe89. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.720144 10807 tablet_service.cc:1023] Processing CreateTablet for tablet 4dfbf2f060fb481fb74acc019b6eaebc (table=rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.720312 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4dfbf2f060fb481fb74acc019b6eaebc. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.720605 10937 tablet_service.cc:1023] Processing CreateTablet for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 (table=rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.720605 11180 tablet_service.cc:1023] Processing CreateTablet for tablet 4dfbf2f060fb481fb74acc019b6eaebc (table=rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.720615 11060 tablet_service.cc:1023] Processing CreateTablet for tablet 4dfbf2f060fb481fb74acc019b6eaebc (table=rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.720685 11180 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4dfbf2f060fb481fb74acc019b6eaebc. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.720685 10937 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.720693 11060 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4dfbf2f060fb481fb74acc019b6eaebc. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.720942 10804 tablet_service.cc:1023] Processing CreateTablet for tablet 4ecac46041da45aba6e82df02354fe89 (table=rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.721027 10804 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ecac46041da45aba6e82df02354fe89. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.721309 10805 tablet_service.cc:1023] Processing CreateTablet for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 (table=rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.721410 10805 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.724771 11765 tablet_bootstrap.cc:439] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:35.725224 11765 tablet_bootstrap.cc:590] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:35.725487 11765 log.cc:810] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b: Log is configured to *not* fsync() on all Append() calls I1121 10:21:35.725740 11766 tablet_bootstrap.cc:439] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:35.726102 11766 tablet_bootstrap.cc:590] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:35.726389 11766 log.cc:810] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6: Log is configured to *not* fsync() on all Append() calls I1121 10:21:35.726598 11767 tablet_bootstrap.cc:439] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:35.726936 11767 tablet_bootstrap.cc:590] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:35.727113 11767 log.cc:810] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794: Log is configured to *not* fsync() on all Append() calls I1121 10:21:35.727319 11768 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:35.727579 11767 tablet_bootstrap.cc:439] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:35.727643 11767 ts_tablet_manager.cc:1104] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:35.727713 11768 tablet_bootstrap.cc:590] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:35.727919 11768 log.cc:810] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: Log is configured to *not* fsync() on all Append() calls I1121 10:21:35.728480 11768 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:35.728612 11768 ts_tablet_manager.cc:1104] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:35.728690 11766 tablet_bootstrap.cc:439] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:35.728792 11765 tablet_bootstrap.cc:439] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:35.728799 11766 ts_tablet_manager.cc:1104] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:35.728894 11765 ts_tablet_manager.cc:1104] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s I1121 10:21:35.729346 11767 raft_consensus.cc:337] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.729490 11767 raft_consensus.cc:363] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.729562 11767 raft_consensus.cc:702] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:35.729784 11767 consensus_queue.cc:231] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.730062 11767 ts_tablet_manager.cc:1132] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I1121 10:21:35.730155 11767 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:35.730391 11768 raft_consensus.cc:337] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.730505 11768 raft_consensus.cc:363] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.730554 11767 tablet_bootstrap.cc:590] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:35.730576 11768 raft_consensus.cc:702] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:35.730641 11765 raft_consensus.cc:337] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:35.730752 11768 consensus_queue.cc:231] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.730834 11765 raft_consensus.cc:363] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.730911 11765 raft_consensus.cc:702] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:35.731279 11768 ts_tablet_manager.cc:1132] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s I1121 10:21:35.731293 11767 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:35.731323 11767 ts_tablet_manager.cc:1104] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:35.731389 11768 tablet_bootstrap.cc:439] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:35.731446 11766 raft_consensus.cc:337] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:35.731540 11765 consensus_queue.cc:231] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:35.731547 11766 raft_consensus.cc:363] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.731570 11766 raft_consensus.cc:702] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:35.731726 11766 consensus_queue.cc:231] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:35.731765 11767 raft_consensus.cc:337] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.731781 11767 raft_consensus.cc:363] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.731798 11767 raft_consensus.cc:702] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:35.731822 11768 tablet_bootstrap.cc:590] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:35.731928 11766 ts_tablet_manager.cc:1132] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s I1121 10:21:35.732026 11766 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:35.732266 11765 ts_tablet_manager.cc:1132] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s I1121 10:21:35.732565 11767 consensus_queue.cc:231] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.732643 11767 ts_tablet_manager.cc:1132] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:35.732841 11768 tablet_bootstrap.cc:439] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:35.732970 11765 tablet_bootstrap.cc:439] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:35.733024 11776 raft_consensus.cc:469] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:35.733129 11776 raft_consensus.cc:491] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:35.733176 11766 tablet_bootstrap.cc:590] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:35.733378 11776 leader_election.cc:296] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:35.733765 11765 tablet_bootstrap.cc:590] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:35.734036 11768 ts_tablet_manager.cc:1104] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:35.734236 11768 raft_consensus.cc:337] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.734331 11767 tablet_bootstrap.cc:439] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:35.734674 11767 tablet_bootstrap.cc:590] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:35.734009 11766 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:35.734982 11768 raft_consensus.cc:363] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.734995 11768 raft_consensus.cc:702] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:35.735064 11766 ts_tablet_manager.cc:1104] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:35.735088 11768 consensus_queue.cc:231] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.735224 11768 ts_tablet_manager.cc:1132] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:35.735273 11766 raft_consensus.cc:337] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.735345 11766 raft_consensus.cc:363] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.735397 11766 raft_consensus.cc:702] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:35.735496 11765 tablet_bootstrap.cc:439] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:35.735548 11766 consensus_queue.cc:231] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.735574 11765 ts_tablet_manager.cc:1104] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:35.735663 11766 ts_tablet_manager.cc:1132] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:35.735905 11765 raft_consensus.cc:337] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.736060 11765 raft_consensus.cc:363] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.736192 11775 raft_consensus.cc:469] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:35.736212 11767 tablet_bootstrap.cc:439] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:35.736240 11767 ts_tablet_manager.cc:1104] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:35.736352 11775 raft_consensus.cc:491] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.736459 11767 raft_consensus.cc:337] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:35.736487 11767 raft_consensus.cc:363] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.736546 11767 raft_consensus.cc:702] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:35.736624 11767 consensus_queue.cc:231] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:35.736699 11767 ts_tablet_manager.cc:1132] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s I1121 10:21:35.737386 11775 leader_election.cc:296] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:35.737480 11765 raft_consensus.cc:702] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:35.737884 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4dfbf2f060fb481fb74acc019b6eaebc" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:35.737953 11205 raft_consensus.cc:2359] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:35.738023 11765 consensus_queue.cc:231] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.738063 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4dfbf2f060fb481fb74acc019b6eaebc" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:35.738117 10816 raft_consensus.cc:2359] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:35.738173 11014 leader_election.cc:310] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:35.738261 11776 raft_consensus.cc:2653] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:35.738292 11776 raft_consensus.cc:469] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:35.738303 11776 raft_consensus.cc:2905] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.738780 11765 ts_tablet_manager.cc:1132] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s I1121 10:21:35.739135 11776 raft_consensus.cc:491] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:35.739346 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4dfbf2f060fb481fb74acc019b6eaebc" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:35.739382 11205 raft_consensus.cc:2905] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.740072 11205 raft_consensus.cc:2359] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:35.740268 11014 leader_election.cc:310] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:35.740336 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4dfbf2f060fb481fb74acc019b6eaebc" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:35.740375 10816 raft_consensus.cc:2905] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.740422 11788 raft_consensus.cc:2653] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:35.740589 11788 raft_consensus.cc:664] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:35.740687 11788 consensus_queue.cc:208] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:35.741168 10816 raft_consensus.cc:2359] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:35.741312 11776 leader_election.cc:296] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:35.741506 10710 catalog_manager.cc:4143] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } } I1121 10:21:35.741511 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8bbbf8a8d2df4f76aa83b5fe842b29a6" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:35.741557 11205 raft_consensus.cc:2359] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:35.741578 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8bbbf8a8d2df4f76aa83b5fe842b29a6" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:35.741607 10816 raft_consensus.cc:2359] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:35.741703 10890 leader_election.cc:310] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:35.741819 11775 raft_consensus.cc:2653] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:35.741853 11775 raft_consensus.cc:469] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:35.741863 11775 raft_consensus.cc:2905] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.742509 11775 raft_consensus.cc:491] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.742573 11775 leader_election.cc:296] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Requested vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:35.742774 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8bbbf8a8d2df4f76aa83b5fe842b29a6" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:35.742774 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8bbbf8a8d2df4f76aa83b5fe842b29a6" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:35.742815 10816 raft_consensus.cc:2905] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.742815 11205 raft_consensus.cc:2905] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.743317 10816 raft_consensus.cc:2359] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:35.743355 11205 raft_consensus.cc:2359] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:35.743489 10887 leader_election.cc:310] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:35.743564 11775 raft_consensus.cc:2653] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:35.743657 11775 raft_consensus.cc:664] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:35.743790 11775 consensus_queue.cc:208] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.744316 10710 catalog_manager.cc:4143] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 0 to 1, leader changed from to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } } W1121 10:21:35.745947 11249 tablet.cc:1767] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:35.768430 11775 raft_consensus.cc:469] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:35.768591 11775 raft_consensus.cc:491] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.768688 11775 leader_election.cc:296] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:35.769124 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:35.769176 10816 raft_consensus.cc:2359] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:35.769328 10887 leader_election.cc:310] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:35.769498 11775 raft_consensus.cc:2653] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:35.769524 11775 raft_consensus.cc:469] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:35.769533 11775 raft_consensus.cc:2905] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.770318 11775 raft_consensus.cc:491] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.770385 11775 leader_election.cc:296] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Requested vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:35.770617 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:35.770646 10816 raft_consensus.cc:2905] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.771351 10816 raft_consensus.cc:2359] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:35.771515 10887 leader_election.cc:310] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:35.771584 11775 raft_consensus.cc:2653] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:35.771620 11775 raft_consensus.cc:664] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:35.771739 11775 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.772006 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:35.772074 11080 raft_consensus.cc:2359] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:35.772006 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:35.772150 11070 raft_consensus.cc:2905] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.772367 10710 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 0 to 1, leader changed from to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } } I1121 10:21:35.772876 11070 raft_consensus.cc:2359] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. W1121 10:21:35.807571 11001 tablet.cc:1767] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: Can't schedule compaction. Clean time has not been advanced past its initial value. W1121 10:21:35.831578 11794 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_0' I1121 10:21:35.861132 11205 raft_consensus.cc:1181] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:35.861181 10816 raft_consensus.cc:1181] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:35.861503 11791 consensus_queue.cc:984] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:35.861737 11775 consensus_queue.cc:984] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:35.862396 10821 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:35.862574 11070 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:35.862716 11791 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:35.862823 11775 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:35.865538 11810 mvcc.cc:203] Tried to move safe_time back from 6448462626248269824 to 6448462625766891520. Current Snapshot: MvccSnapshot[committed={T|T < 6448462626244890624}] I1121 10:21:35.867624 11818 mvcc.cc:203] Tried to move safe_time back from 6448462626248871936 to 6448462625881387008. Current Snapshot: MvccSnapshot[committed={T|T < 6448462626248871936}] I1121 10:21:35.868424 11205 raft_consensus.cc:1181] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:35.868729 11776 consensus_queue.cc:984] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:35.868788 10821 raft_consensus.cc:1181] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:35.869148 11776 consensus_queue.cc:984] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:35.871188 11816 mvcc.cc:203] Tried to move safe_time back from 6448462626248871936 to 6448462625881387008. Current Snapshot: MvccSnapshot[committed={T|T < 6448462626248871936}] I1121 10:21:35.873152 11827 mvcc.cc:203] Tried to move safe_time back from 6448462626250768384 to 6448462625754763264. Current Snapshot: MvccSnapshot[committed={T|T < 6448462626250768384}] Generator report time total : 29.2207 ms time per row: 0.0146104 ms I1121 10:21:35.880416 10705 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:35.882115 11176 tablet_service.cc:1023] Processing CreateTablet for tablet 69c25581de6e4945a167245a4dcbbfe0 (table=rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.882135 10807 tablet_service.cc:1023] Processing CreateTablet for tablet 51d8830759cc4842af19cc0d22bb8d65 (table=rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.882184 10937 tablet_service.cc:1023] Processing CreateTablet for tablet 69c25581de6e4945a167245a4dcbbfe0 (table=rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.882262 11176 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69c25581de6e4945a167245a4dcbbfe0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.882285 10937 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69c25581de6e4945a167245a4dcbbfe0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.882285 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 51d8830759cc4842af19cc0d22bb8d65. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.882122 11180 tablet_service.cc:1023] Processing CreateTablet for tablet 7b34e50304ff46e59622de2e979ac7b8 (table=rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.882762 10936 tablet_service.cc:1023] Processing CreateTablet for tablet 7b34e50304ff46e59622de2e979ac7b8 (table=rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.882820 11180 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7b34e50304ff46e59622de2e979ac7b8. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.882835 10936 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7b34e50304ff46e59622de2e979ac7b8. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.883499 11767 tablet_bootstrap.cc:439] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:35.883752 11765 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:35.883842 11767 tablet_bootstrap.cc:590] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:35.883962 10935 tablet_service.cc:1023] Processing CreateTablet for tablet 51d8830759cc4842af19cc0d22bb8d65 (table=rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.884037 10935 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 51d8830759cc4842af19cc0d22bb8d65. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.884083 11765 tablet_bootstrap.cc:590] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:35.884425 11061 tablet_service.cc:1023] Processing CreateTablet for tablet 69c25581de6e4945a167245a4dcbbfe0 (table=rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.884577 11768 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:35.884593 11060 tablet_service.cc:1023] Processing CreateTablet for tablet 7b34e50304ff46e59622de2e979ac7b8 (table=rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.884755 11059 tablet_service.cc:1023] Processing CreateTablet for tablet 51d8830759cc4842af19cc0d22bb8d65 (table=rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:35.884898 11061 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69c25581de6e4945a167245a4dcbbfe0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.884910 11768 tablet_bootstrap.cc:590] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:35.886081 11766 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:35.886106 11060 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7b34e50304ff46e59622de2e979ac7b8. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.886394 11766 tablet_bootstrap.cc:590] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:35.886981 11767 tablet_bootstrap.cc:439] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:35.887025 11767 ts_tablet_manager.cc:1104] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s I1121 10:21:35.887049 11765 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:35.887084 11765 ts_tablet_manager.cc:1104] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:35.887198 11767 raft_consensus.cc:337] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.887213 11767 raft_consensus.cc:363] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.887220 11767 raft_consensus.cc:702] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:35.887225 11765 raft_consensus.cc:337] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.887238 11765 raft_consensus.cc:363] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.887256 11765 raft_consensus.cc:702] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:35.887270 11059 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 51d8830759cc4842af19cc0d22bb8d65. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:35.887291 11767 consensus_queue.cc:231] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.887321 11765 consensus_queue.cc:231] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.887364 11767 ts_tablet_manager.cc:1132] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:35.887384 11765 ts_tablet_manager.cc:1132] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:35.887416 11765 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:35.887415 11768 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:35.887446 11768 ts_tablet_manager.cc:1104] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:35.887588 11768 raft_consensus.cc:337] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.887603 11768 raft_consensus.cc:363] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.887609 11768 raft_consensus.cc:702] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:35.887665 11768 consensus_queue.cc:231] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.887722 11765 tablet_bootstrap.cc:590] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:35.887742 11768 ts_tablet_manager.cc:1132] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:35.887794 11768 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:35.888072 11768 tablet_bootstrap.cc:590] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:35.887274 11766 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:35.888572 11766 ts_tablet_manager.cc:1104] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:35.888721 11766 raft_consensus.cc:337] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.888743 11766 raft_consensus.cc:363] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.888751 11766 raft_consensus.cc:702] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:35.888809 11766 consensus_queue.cc:231] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.888873 11766 ts_tablet_manager.cc:1132] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:35.888906 11766 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:35.888942 11768 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:35.888967 11768 ts_tablet_manager.cc:1104] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:35.889115 11768 raft_consensus.cc:337] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.889148 11768 raft_consensus.cc:363] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.889156 11768 raft_consensus.cc:702] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:35.889225 11768 consensus_queue.cc:231] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.889240 11766 tablet_bootstrap.cc:590] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:35.889290 11768 ts_tablet_manager.cc:1132] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:35.889340 11768 tablet_bootstrap.cc:439] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:35.889660 11768 tablet_bootstrap.cc:590] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:35.890318 11766 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:35.890354 11766 ts_tablet_manager.cc:1104] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:35.890417 11768 tablet_bootstrap.cc:439] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:35.890417 11765 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:35.890444 11768 ts_tablet_manager.cc:1104] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:35.890444 11765 ts_tablet_manager.cc:1104] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:35.890483 11766 raft_consensus.cc:337] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.890494 11766 raft_consensus.cc:363] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.890498 11766 raft_consensus.cc:702] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:35.890555 11766 consensus_queue.cc:231] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.890573 11768 raft_consensus.cc:337] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.890588 11768 raft_consensus.cc:363] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.890586 11765 raft_consensus.cc:337] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.890595 11768 raft_consensus.cc:702] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:35.890601 11765 raft_consensus.cc:363] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.890609 11765 raft_consensus.cc:702] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:35.890614 11766 ts_tablet_manager.cc:1132] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:35.890650 11766 tablet_bootstrap.cc:439] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:35.890661 11765 consensus_queue.cc:231] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.890661 11768 consensus_queue.cc:231] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.890723 11768 ts_tablet_manager.cc:1132] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:35.890729 11765 ts_tablet_manager.cc:1132] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:35.891016 11766 tablet_bootstrap.cc:590] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:35.891526 11766 tablet_bootstrap.cc:439] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:35.891567 11766 ts_tablet_manager.cc:1104] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:35.891711 11766 raft_consensus.cc:337] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.891733 11766 raft_consensus.cc:363] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:35.891739 11766 raft_consensus.cc:702] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:35.891813 11766 consensus_queue.cc:231] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.891906 11766 ts_tablet_manager.cc:1132] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:35.901260 11823 raft_consensus.cc:469] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:35.901356 11823 raft_consensus.cc:491] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.901479 11823 leader_election.cc:296] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:35.901684 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "69c25581de6e4945a167245a4dcbbfe0" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:35.901687 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "69c25581de6e4945a167245a4dcbbfe0" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:35.901724 11205 raft_consensus.cc:2359] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:35.901726 11070 raft_consensus.cc:2359] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:35.901896 10887 leader_election.cc:310] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:35.901983 11823 raft_consensus.cc:2653] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:35.902004 11823 raft_consensus.cc:469] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:35.902010 11823 raft_consensus.cc:2905] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.902624 11823 raft_consensus.cc:491] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.902694 11823 leader_election.cc:296] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Requested vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:35.902861 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "69c25581de6e4945a167245a4dcbbfe0" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:35.902889 11070 raft_consensus.cc:2905] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.902894 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "69c25581de6e4945a167245a4dcbbfe0" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:35.902914 11205 raft_consensus.cc:2905] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.903404 11070 raft_consensus.cc:2359] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:35.903429 11205 raft_consensus.cc:2359] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:35.903551 10887 leader_election.cc:310] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:35.903625 11823 raft_consensus.cc:2653] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:35.903662 11823 raft_consensus.cc:664] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:35.903723 11823 consensus_queue.cc:208] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:35.904304 10707 catalog_manager.cc:4143] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 0 to 1, leader changed from to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } } W1121 10:21:35.904489 11125 tablet.cc:1767] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:35.953845 11789 raft_consensus.cc:469] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:35.953986 11789 raft_consensus.cc:491] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } W1121 10:21:35.953997 10876 tablet.cc:1767] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:21:35.954133 11789 leader_election.cc:296] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:35.954423 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "51d8830759cc4842af19cc0d22bb8d65" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:35.954473 10821 raft_consensus.cc:2359] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:35.954632 11011 leader_election.cc:310] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:35.954792 11789 raft_consensus.cc:2653] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:35.954818 11789 raft_consensus.cc:469] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:35.954828 11789 raft_consensus.cc:2905] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.955756 11789 raft_consensus.cc:491] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.955828 11789 leader_election.cc:296] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:35.955993 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "51d8830759cc4842af19cc0d22bb8d65" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:35.956024 10821 raft_consensus.cc:2905] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.956827 10821 raft_consensus.cc:2359] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:35.956990 11011 leader_election.cc:310] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:35.957073 11789 raft_consensus.cc:2653] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:35.957120 11789 raft_consensus.cc:664] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:35.957195 11789 consensus_queue.cc:208] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:35.957638 10955 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "51d8830759cc4842af19cc0d22bb8d65" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:35.957695 10955 raft_consensus.cc:2359] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:35.957845 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "51d8830759cc4842af19cc0d22bb8d65" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:35.957870 10953 raft_consensus.cc:2905] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:35.957897 10707 catalog_manager.cc:4143] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } } I1121 10:21:35.958633 10953 raft_consensus.cc:2359] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:36.161551 11789 raft_consensus.cc:469] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:36.161649 11789 raft_consensus.cc:491] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.161831 11789 leader_election.cc:296] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:36.162089 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:36.162108 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:36.162135 10953 raft_consensus.cc:2359] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:36.162140 11205 raft_consensus.cc:2359] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:36.162313 11013 leader_election.cc:310] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:36.162420 11789 raft_consensus.cc:2653] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:36.162446 11789 raft_consensus.cc:469] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:36.162454 11789 raft_consensus.cc:2905] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.163062 11789 raft_consensus.cc:491] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.163153 11789 leader_election.cc:296] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:36.163370 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:36.163403 11205 raft_consensus.cc:2905] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.163415 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:36.163450 10953 raft_consensus.cc:2905] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.163933 11205 raft_consensus.cc:2359] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:36.164070 10953 raft_consensus.cc:2359] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:36.164098 11014 leader_election.cc:310] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:36.164165 11789 raft_consensus.cc:2653] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:36.164273 11789 raft_consensus.cc:664] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:36.164363 11789 consensus_queue.cc:208] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.165083 10707 catalog_manager.cc:4143] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } } W1121 10:21:36.291188 11847 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_1' I1121 10:21:36.340962 10821 raft_consensus.cc:1181] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.341329 11776 consensus_queue.cc:984] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.341508 11205 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.341941 11776 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.341945 10953 raft_consensus.cc:1181] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.342183 10955 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.343163 11070 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.343645 11822 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.345149 11204 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.345531 11789 consensus_queue.cc:984] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.345820 11789 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.347239 11823 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 53.5271 ms time per row: 0.0267636 ms I1121 10:21:36.378973 10705 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:36.382591 10935 tablet_service.cc:1023] Processing CreateTablet for tablet c7de6e746a8246f89d8b65b59803c3af (table=rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.382771 11180 tablet_service.cc:1023] Processing CreateTablet for tablet 702635e315b940138670eeb5ae6ace1b (table=rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.382805 11176 tablet_service.cc:1023] Processing CreateTablet for tablet c7de6e746a8246f89d8b65b59803c3af (table=rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.382936 10807 tablet_service.cc:1023] Processing CreateTablet for tablet c7de6e746a8246f89d8b65b59803c3af (table=rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.383038 10935 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c7de6e746a8246f89d8b65b59803c3af. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.383085 11185 tablet_service.cc:1023] Processing CreateTablet for tablet cc68d4b786cd485285707592d3c25f80 (table=rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.383164 11180 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 702635e315b940138670eeb5ae6ace1b. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.383246 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c7de6e746a8246f89d8b65b59803c3af. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.383350 11185 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet cc68d4b786cd485285707592d3c25f80. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.385160 10937 tablet_service.cc:1023] Processing CreateTablet for tablet cc68d4b786cd485285707592d3c25f80 (table=rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.385433 10937 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet cc68d4b786cd485285707592d3c25f80. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.386811 11176 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c7de6e746a8246f89d8b65b59803c3af. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.387609 11767 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:36.388231 11767 tablet_bootstrap.cc:590] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:36.388676 11768 tablet_bootstrap.cc:439] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:36.389650 11768 tablet_bootstrap.cc:590] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:36.389804 11059 tablet_service.cc:1023] Processing CreateTablet for tablet 702635e315b940138670eeb5ae6ace1b (table=rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.390018 11059 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 702635e315b940138670eeb5ae6ace1b. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.391523 11765 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:36.391902 11768 tablet_bootstrap.cc:439] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:36.391983 11768 ts_tablet_manager.cc:1104] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s I1121 10:21:36.392560 11768 raft_consensus.cc:337] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:36.392594 11768 raft_consensus.cc:363] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.392630 11768 raft_consensus.cc:702] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:36.392802 11765 tablet_bootstrap.cc:590] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:36.392844 11768 consensus_queue.cc:231] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:36.393097 11768 ts_tablet_manager.cc:1132] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:36.393164 11768 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:36.394110 10807 tablet_service.cc:1023] Processing CreateTablet for tablet 702635e315b940138670eeb5ae6ace1b (table=rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.394235 11768 tablet_bootstrap.cc:590] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:36.394351 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 702635e315b940138670eeb5ae6ace1b. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.395052 11873 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:36.396160 11873 tablet_bootstrap.cc:590] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:36.397379 11765 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:36.397465 11765 ts_tablet_manager.cc:1104] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s I1121 10:21:36.397891 11767 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:36.397985 11767 ts_tablet_manager.cc:1104] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s I1121 10:21:36.398027 11765 raft_consensus.cc:337] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.398043 11765 raft_consensus.cc:363] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.398051 11765 raft_consensus.cc:702] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:36.398159 10804 tablet_service.cc:1023] Processing CreateTablet for tablet cc68d4b786cd485285707592d3c25f80 (table=rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.398334 11767 raft_consensus.cc:337] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.398386 10804 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet cc68d4b786cd485285707592d3c25f80. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.398403 11767 raft_consensus.cc:363] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.398438 11767 raft_consensus.cc:702] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:36.398605 11767 consensus_queue.cc:231] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.398869 11767 ts_tablet_manager.cc:1132] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:36.399036 11767 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:36.399349 11765 consensus_queue.cc:231] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.399900 11767 tablet_bootstrap.cc:590] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:36.400393 11765 ts_tablet_manager.cc:1132] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s I1121 10:21:36.401790 11768 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:36.401885 11768 ts_tablet_manager.cc:1104] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s I1121 10:21:36.402392 11768 raft_consensus.cc:337] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.402431 11768 raft_consensus.cc:363] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.402456 11768 raft_consensus.cc:702] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:36.402798 11768 consensus_queue.cc:231] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.403091 11768 ts_tablet_manager.cc:1132] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:36.403558 11767 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:36.401021 11765 tablet_bootstrap.cc:439] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:36.403676 11767 ts_tablet_manager.cc:1104] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s I1121 10:21:36.404214 11767 raft_consensus.cc:337] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.404232 11767 raft_consensus.cc:363] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.404239 11767 raft_consensus.cc:702] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:36.404507 11767 consensus_queue.cc:231] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.404633 11765 tablet_bootstrap.cc:590] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:36.404773 11767 ts_tablet_manager.cc:1132] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:36.404868 11767 tablet_bootstrap.cc:439] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:36.405690 11829 raft_consensus.cc:469] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:36.405956 11829 raft_consensus.cc:491] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.406296 11767 tablet_bootstrap.cc:590] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:36.406663 11829 leader_election.cc:296] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:36.408469 11767 tablet_bootstrap.cc:439] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:36.408469 11765 tablet_bootstrap.cc:439] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:36.408717 11767 ts_tablet_manager.cc:1104] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s I1121 10:21:36.408740 11765 ts_tablet_manager.cc:1104] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s I1121 10:21:36.409047 11767 raft_consensus.cc:337] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:36.409051 11765 raft_consensus.cc:337] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:36.409103 11767 raft_consensus.cc:363] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.409103 11765 raft_consensus.cc:363] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.409147 11767 raft_consensus.cc:702] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:36.409147 11765 raft_consensus.cc:702] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:36.409368 11765 consensus_queue.cc:231] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:36.409368 11767 consensus_queue.cc:231] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:36.411197 11765 ts_tablet_manager.cc:1132] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:36.411206 11767 ts_tablet_manager.cc:1132] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:36.411270 11765 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:36.411666 11765 tablet_bootstrap.cc:590] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:36.414499 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:36.414640 11204 raft_consensus.cc:2359] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 0. I1121 10:21:36.414815 11777 raft_consensus.cc:469] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:36.414983 10765 leader_election.cc:310] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:36.415079 11777 raft_consensus.cc:491] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:36.415345 11829 raft_consensus.cc:2653] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:36.415436 11829 raft_consensus.cc:469] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:36.415450 11829 raft_consensus.cc:2905] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.415505 11777 leader_election.cc:296] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:36.415724 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true W1121 10:21:36.416298 10762 leader_election.cc:348] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): Illegal state: must be running to vote when last-logged opid is not known I1121 10:21:36.417742 11829 raft_consensus.cc:491] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.417937 11829 leader_election.cc:296] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Requested vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:36.418262 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:36.418300 11204 raft_consensus.cc:2905] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.418319 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:36.418412 11765 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:36.418450 11765 ts_tablet_manager.cc:1104] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s W1121 10:21:36.418514 10762 leader_election.cc:348] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): Illegal state: must be running to vote when last-logged opid is not known I1121 10:21:36.418757 11765 raft_consensus.cc:337] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.418782 11765 raft_consensus.cc:363] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.418797 11765 raft_consensus.cc:702] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:36.418982 11765 consensus_queue.cc:231] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.419147 11765 ts_tablet_manager.cc:1132] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:36.419776 11204 raft_consensus.cc:2359] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 1. I1121 10:21:36.420159 10765 leader_election.cc:310] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: ca8a596d4c4a4b438860e5e0420e3ad6 I1121 10:21:36.420280 11829 raft_consensus.cc:2653] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:36.420428 11829 raft_consensus.cc:664] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Becoming Leader. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Running, Role: LEADER I1121 10:21:36.420641 11829 consensus_queue.cc:208] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.422004 10710 catalog_manager.cc:4143] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 reported cstate change: term changed from 0 to 1, leader changed from to cd693d36f0014dc3a7c3f26196124794 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } } I1121 10:21:36.424468 11873 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:36.424571 11873 ts_tablet_manager.cc:1104] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.001s I1121 10:21:36.424856 11873 raft_consensus.cc:337] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.424896 11873 raft_consensus.cc:363] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.424909 11873 raft_consensus.cc:702] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:36.425102 11873 consensus_queue.cc:231] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.425493 11873 ts_tablet_manager.cc:1132] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:36.428580 10955 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "cc68d4b786cd485285707592d3c25f80" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:36.428663 10955 raft_consensus.cc:2359] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:36.428973 11137 leader_election.cc:310] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:36.429180 11777 raft_consensus.cc:2653] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:36.429234 11777 raft_consensus.cc:469] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:36.429244 11777 raft_consensus.cc:2905] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.429770 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "cc68d4b786cd485285707592d3c25f80" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:36.429904 10821 raft_consensus.cc:2359] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:36.430666 11777 raft_consensus.cc:491] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:36.430778 11777 leader_election.cc:296] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:36.431068 10955 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "cc68d4b786cd485285707592d3c25f80" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:36.431072 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "cc68d4b786cd485285707592d3c25f80" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:36.431102 10821 raft_consensus.cc:2905] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.431111 10955 raft_consensus.cc:2905] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.431773 10821 raft_consensus.cc:2359] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:36.432093 11135 leader_election.cc:310] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:36.432101 10955 raft_consensus.cc:2359] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:36.432193 11777 raft_consensus.cc:2653] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:36.432350 11777 raft_consensus.cc:664] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Becoming Leader. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Running, Role: LEADER I1121 10:21:36.432452 11777 consensus_queue.cc:208] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:36.433778 10710 catalog_manager.cc:4143] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: term changed from 0 to 1, leader changed from to 5457a73f9ecb40ee9889513a8412937b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } } I1121 10:21:36.537178 11887 raft_consensus.cc:469] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:36.537362 11887 raft_consensus.cc:491] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.537597 11887 leader_election.cc:296] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:36.538203 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c7de6e746a8246f89d8b65b59803c3af" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:36.538321 11204 raft_consensus.cc:2359] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 0. I1121 10:21:36.538688 10765 leader_election.cc:310] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:36.538830 11887 raft_consensus.cc:2653] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:36.538920 11887 raft_consensus.cc:469] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:36.538959 11887 raft_consensus.cc:2905] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.540388 11887 raft_consensus.cc:491] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.540539 11887 leader_election.cc:296] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:36.541007 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c7de6e746a8246f89d8b65b59803c3af" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:36.541043 11204 raft_consensus.cc:2905] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.542068 11204 raft_consensus.cc:2359] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 1. I1121 10:21:36.542275 10765 leader_election.cc:310] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:36.542372 11887 raft_consensus.cc:2653] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:36.542490 11887 raft_consensus.cc:664] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Becoming Leader. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Running, Role: LEADER I1121 10:21:36.542651 11887 consensus_queue.cc:208] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.542800 10955 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c7de6e746a8246f89d8b65b59803c3af" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:36.542873 10955 raft_consensus.cc:2359] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 0. I1121 10:21:36.542868 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "c7de6e746a8246f89d8b65b59803c3af" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:36.542961 10953 raft_consensus.cc:2905] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.543740 10710 catalog_manager.cc:4143] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 reported cstate change: term changed from 0 to 1, leader changed from to cd693d36f0014dc3a7c3f26196124794 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } } I1121 10:21:36.543911 10953 raft_consensus.cc:2359] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 1. I1121 10:21:36.554039 11861 raft_consensus.cc:469] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:36.554237 11861 raft_consensus.cc:491] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:36.554467 11861 leader_election.cc:296] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:36.554927 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:36.555039 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:36.555085 11204 raft_consensus.cc:2289] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca8a596d4c4a4b438860e5e0420e3ad6 in current term 1: Already voted for candidate cd693d36f0014dc3a7c3f26196124794 in this term. I1121 10:21:36.555160 10821 raft_consensus.cc:2320] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate ca8a596d4c4a4b438860e5e0420e3ad6 for term 1 because replica is either leader or believes a valid leader to be alive. I1121 10:21:36.555577 11014 leader_election.cc:310] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6; no voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794 I1121 10:21:36.555837 11861 raft_consensus.cc:2905] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.557631 11861 raft_consensus.cc:2601] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1121 10:21:36.650569 11892 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_2' I1121 10:21:36.678264 10821 raft_consensus.cc:1181] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.678557 11777 consensus_queue.cc:984] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.678674 10953 raft_consensus.cc:1181] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:36.678937 11777 consensus_queue.cc:984] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.679306 11204 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.679337 11070 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.679383 11205 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.679579 11829 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.680194 11829 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.682557 11829 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:36.682868 10953 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:36.683094 11908 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 23.0809 ms time per row: 0.0115405 ms I1121 10:21:36.693315 10710 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:36.694950 10935 tablet_service.cc:1023] Processing CreateTablet for tablet 5becd9fee1eb40cf8eebb4ffac223e15 (table=rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.695055 11185 tablet_service.cc:1023] Processing CreateTablet for tablet 635b29a1f41e418faffe3677886048e5 (table=rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.695070 10935 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5becd9fee1eb40cf8eebb4ffac223e15. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.695158 11185 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 635b29a1f41e418faffe3677886048e5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.695269 11059 tablet_service.cc:1023] Processing CreateTablet for tablet 5becd9fee1eb40cf8eebb4ffac223e15 (table=rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.695288 10937 tablet_service.cc:1023] Processing CreateTablet for tablet 635b29a1f41e418faffe3677886048e5 (table=rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.695355 11059 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5becd9fee1eb40cf8eebb4ffac223e15. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.695366 10937 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 635b29a1f41e418faffe3677886048e5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.696297 11768 tablet_bootstrap.cc:439] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:36.696357 11765 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:36.695055 11176 tablet_service.cc:1023] Processing CreateTablet for tablet 8963dd2bc2da4cacb9d5e5329680d7d7 (table=rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.696594 11060 tablet_service.cc:1023] Processing CreateTablet for tablet 635b29a1f41e418faffe3677886048e5 (table=rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.696612 11176 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8963dd2bc2da4cacb9d5e5329680d7d7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.696595 11873 tablet_bootstrap.cc:439] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:36.696684 11060 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 635b29a1f41e418faffe3677886048e5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.696939 11873 tablet_bootstrap.cc:590] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:36.696940 11768 tablet_bootstrap.cc:590] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:36.697748 11061 tablet_service.cc:1023] Processing CreateTablet for tablet 8963dd2bc2da4cacb9d5e5329680d7d7 (table=rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.697754 11765 tablet_bootstrap.cc:590] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:36.697830 11061 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8963dd2bc2da4cacb9d5e5329680d7d7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.698242 11873 tablet_bootstrap.cc:439] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:36.698283 11873 ts_tablet_manager.cc:1104] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:36.698354 11768 tablet_bootstrap.cc:439] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:36.698382 11768 ts_tablet_manager.cc:1104] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:36.698446 11873 raft_consensus.cc:337] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.698460 11873 raft_consensus.cc:363] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.698462 11765 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:36.698469 11873 raft_consensus.cc:702] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:36.698488 11765 ts_tablet_manager.cc:1104] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:36.698520 11873 consensus_queue.cc:231] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.698519 11768 raft_consensus.cc:337] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.698534 11768 raft_consensus.cc:363] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.698540 11768 raft_consensus.cc:702] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:36.698591 11873 ts_tablet_manager.cc:1132] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:36.698606 11768 consensus_queue.cc:231] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.698624 11873 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:36.698626 11765 raft_consensus.cc:337] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:36.698639 11765 raft_consensus.cc:363] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.698647 11765 raft_consensus.cc:702] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:36.698678 11768 ts_tablet_manager.cc:1132] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:36.698710 11765 consensus_queue.cc:231] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:36.698722 11768 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:36.698724 10807 tablet_service.cc:1023] Processing CreateTablet for tablet 5becd9fee1eb40cf8eebb4ffac223e15 (table=rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.698791 11765 ts_tablet_manager.cc:1132] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:36.698825 11765 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:36.698829 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5becd9fee1eb40cf8eebb4ffac223e15. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.698920 11873 tablet_bootstrap.cc:590] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:36.699023 11768 tablet_bootstrap.cc:590] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:36.699110 11765 tablet_bootstrap.cc:590] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:36.699966 11873 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:36.699997 11873 ts_tablet_manager.cc:1104] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:36.700105 11873 raft_consensus.cc:337] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:36.700124 11873 raft_consensus.cc:363] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.700130 11873 raft_consensus.cc:702] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:36.700132 11767 tablet_bootstrap.cc:439] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:36.700153 10804 tablet_service.cc:1023] Processing CreateTablet for tablet 8963dd2bc2da4cacb9d5e5329680d7d7 (table=rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:36.700175 11873 consensus_queue.cc:231] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:36.700239 10804 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8963dd2bc2da4cacb9d5e5329680d7d7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:36.700255 11873 ts_tablet_manager.cc:1132] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:36.700290 11873 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:36.700465 11767 tablet_bootstrap.cc:590] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:36.700788 11873 tablet_bootstrap.cc:590] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:36.701329 11765 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:36.701364 11765 ts_tablet_manager.cc:1104] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I1121 10:21:36.701366 11873 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:36.701393 11873 ts_tablet_manager.cc:1104] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:36.701504 11765 raft_consensus.cc:337] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.701532 11765 raft_consensus.cc:363] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.701541 11765 raft_consensus.cc:702] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:36.701547 11873 raft_consensus.cc:337] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.701560 11873 raft_consensus.cc:363] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.701573 11873 raft_consensus.cc:702] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:36.701601 11765 consensus_queue.cc:231] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.701627 11873 consensus_queue.cc:231] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.701658 11765 ts_tablet_manager.cc:1132] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:36.701683 11873 ts_tablet_manager.cc:1132] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:36.702224 11768 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:36.702256 11768 ts_tablet_manager.cc:1104] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s I1121 10:21:36.702386 11768 raft_consensus.cc:337] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:36.702491 11768 raft_consensus.cc:363] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.702497 11768 raft_consensus.cc:702] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:36.702554 11768 consensus_queue.cc:231] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:36.702631 11768 ts_tablet_manager.cc:1132] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:36.702901 11767 tablet_bootstrap.cc:439] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:36.702936 11767 ts_tablet_manager.cc:1104] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:36.703075 11767 raft_consensus.cc:337] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.703088 11767 raft_consensus.cc:363] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.703094 11767 raft_consensus.cc:702] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:36.703141 11767 consensus_queue.cc:231] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.703261 11767 ts_tablet_manager.cc:1132] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:36.703295 11767 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:36.703591 11767 tablet_bootstrap.cc:590] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:36.704066 11767 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:36.704099 11767 ts_tablet_manager.cc:1104] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:36.704213 11767 raft_consensus.cc:337] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.704236 11767 raft_consensus.cc:363] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:36.704241 11767 raft_consensus.cc:702] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:36.704290 11767 consensus_queue.cc:231] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.704354 11767 ts_tablet_manager.cc:1132] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:36.804270 11861 raft_consensus.cc:469] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:36.804389 11861 raft_consensus.cc:491] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.804504 11861 leader_election.cc:296] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:36.804802 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:36.804847 10821 raft_consensus.cc:2359] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:36.804848 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:36.804888 11204 raft_consensus.cc:2359] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:36.804996 11011 leader_election.cc:310] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:36.805244 11861 raft_consensus.cc:2653] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:36.805266 11861 raft_consensus.cc:469] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:36.805275 11861 raft_consensus.cc:2905] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.805917 11861 raft_consensus.cc:491] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.806061 11861 leader_election.cc:296] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:36.806167 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:36.806177 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:36.806222 10821 raft_consensus.cc:2905] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.806226 11204 raft_consensus.cc:2905] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.806895 11204 raft_consensus.cc:2359] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:36.806947 10821 raft_consensus.cc:2359] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:36.807090 11014 leader_election.cc:310] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:36.807176 11861 raft_consensus.cc:2653] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:36.807252 11861 raft_consensus.cc:664] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:36.807330 11861 consensus_queue.cc:208] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.808091 10704 catalog_manager.cc:4143] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } } I1121 10:21:36.837684 11861 raft_consensus.cc:469] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:36.837781 11861 raft_consensus.cc:491] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:36.837913 11861 leader_election.cc:296] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:36.838089 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "635b29a1f41e418faffe3677886048e5" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:36.838132 11204 raft_consensus.cc:2359] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:36.838254 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "635b29a1f41e418faffe3677886048e5" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:36.838289 11014 leader_election.cc:310] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:36.838299 10953 raft_consensus.cc:2359] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:36.838382 11861 raft_consensus.cc:2653] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:36.838398 11861 raft_consensus.cc:469] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:36.838407 11861 raft_consensus.cc:2905] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.839124 11861 raft_consensus.cc:491] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:36.839201 11861 leader_election.cc:296] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:36.839365 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "635b29a1f41e418faffe3677886048e5" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:36.839368 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "635b29a1f41e418faffe3677886048e5" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:36.839392 11204 raft_consensus.cc:2905] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.839392 10953 raft_consensus.cc:2905] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.839881 11204 raft_consensus.cc:2359] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:36.839939 10953 raft_consensus.cc:2359] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:36.840029 11014 leader_election.cc:310] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:36.840095 11861 raft_consensus.cc:2653] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:36.840122 11861 raft_consensus.cc:664] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:36.840188 11861 consensus_queue.cc:208] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:36.840894 10704 catalog_manager.cc:4143] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } } I1121 10:21:36.917711 11822 raft_consensus.cc:469] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:36.917840 11822 raft_consensus.cc:491] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.917953 11822 leader_election.cc:296] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:36.918164 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5becd9fee1eb40cf8eebb4ffac223e15" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:36.918213 11070 raft_consensus.cc:2359] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:36.918179 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5becd9fee1eb40cf8eebb4ffac223e15" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:36.918277 10821 raft_consensus.cc:2359] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:36.918367 10887 leader_election.cc:310] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:36.918450 11822 raft_consensus.cc:2653] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:36.918481 11822 raft_consensus.cc:469] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:36.918488 11822 raft_consensus.cc:2905] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.919160 11822 raft_consensus.cc:491] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.919241 11822 leader_election.cc:296] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Requested vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:36.919412 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5becd9fee1eb40cf8eebb4ffac223e15" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:36.919472 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5becd9fee1eb40cf8eebb4ffac223e15" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:36.919519 10821 raft_consensus.cc:2905] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.919519 11070 raft_consensus.cc:2905] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:36.920214 11070 raft_consensus.cc:2359] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:36.920214 10821 raft_consensus.cc:2359] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:36.920379 10887 leader_election.cc:310] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:36.920454 11822 raft_consensus.cc:2653] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:36.920573 11822 raft_consensus.cc:664] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:36.920650 11822 consensus_queue.cc:208] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:36.921193 10704 catalog_manager.cc:4143] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 0 to 1, leader changed from to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } } W1121 10:21:36.981197 11933 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_3' I1121 10:21:37.008262 10953 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.008262 11204 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.008891 10821 raft_consensus.cc:1181] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.008934 11204 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.008963 11789 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.009088 11789 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.009114 11948 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.009151 11931 consensus_queue.cc:984] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.009402 11070 raft_consensus.cc:1181] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.009567 10816 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.009884 11822 consensus_queue.cc:984] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.011334 11948 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 22.1256 ms time per row: 0.0110628 ms I1121 10:21:37.022781 10707 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:37.024287 10937 tablet_service.cc:1023] Processing CreateTablet for tablet 204d260bb58b494091ae91cb57a88ca8 (table=rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.024382 11059 tablet_service.cc:1023] Processing CreateTablet for tablet 204d260bb58b494091ae91cb57a88ca8 (table=rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.024410 10937 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 204d260bb58b494091ae91cb57a88ca8. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.024412 11176 tablet_service.cc:1023] Processing CreateTablet for tablet df2330a22c8142ada6d056401d0a064d (table=rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.024590 10935 tablet_service.cc:1023] Processing CreateTablet for tablet df2330a22c8142ada6d056401d0a064d (table=rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.024590 11061 tablet_service.cc:1023] Processing CreateTablet for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 (table=rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.024590 11185 tablet_service.cc:1023] Processing CreateTablet for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 (table=rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.024287 10936 tablet_service.cc:1023] Processing CreateTablet for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 (table=rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.024675 11176 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet df2330a22c8142ada6d056401d0a064d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.024819 10804 tablet_service.cc:1023] Processing CreateTablet for tablet 204d260bb58b494091ae91cb57a88ca8 (table=rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.024924 10804 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 204d260bb58b494091ae91cb57a88ca8. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.025225 10935 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet df2330a22c8142ada6d056401d0a064d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.025337 10936 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.025893 11765 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:37.026113 11185 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.026134 10807 tablet_service.cc:1023] Processing CreateTablet for tablet df2330a22c8142ada6d056401d0a064d (table=rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.026203 11765 tablet_bootstrap.cc:590] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:37.026206 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet df2330a22c8142ada6d056401d0a064d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.026300 11768 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:37.026605 11768 tablet_bootstrap.cc:590] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:37.027225 11767 tablet_bootstrap.cc:439] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:37.027513 11767 tablet_bootstrap.cc:590] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:37.027823 11059 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 204d260bb58b494091ae91cb57a88ca8. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.028362 11765 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:37.028403 11765 ts_tablet_manager.cc:1104] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I1121 10:21:37.028441 11768 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:37.028470 11768 ts_tablet_manager.cc:1104] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:37.028578 11767 tablet_bootstrap.cc:439] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:37.028589 11765 raft_consensus.cc:337] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.028604 11765 raft_consensus.cc:363] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.028611 11765 raft_consensus.cc:702] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:37.028610 11767 ts_tablet_manager.cc:1104] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:37.028673 11765 consensus_queue.cc:231] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.028743 11765 ts_tablet_manager.cc:1132] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s I1121 10:21:37.028765 11767 raft_consensus.cc:337] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.028779 11767 raft_consensus.cc:363] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.028779 11765 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:37.028786 11767 raft_consensus.cc:702] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:37.028842 11767 consensus_queue.cc:231] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.028854 11768 raft_consensus.cc:337] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.028868 11768 raft_consensus.cc:363] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.028877 11768 raft_consensus.cc:702] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:37.028908 11767 ts_tablet_manager.cc:1132] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.028935 11768 consensus_queue.cc:231] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.028940 11767 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:37.029006 11768 ts_tablet_manager.cc:1132] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:37.029044 11768 tablet_bootstrap.cc:439] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:37.029093 11061 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.029091 11765 tablet_bootstrap.cc:590] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:37.029242 11767 tablet_bootstrap.cc:590] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:37.029345 11768 tablet_bootstrap.cc:590] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:37.029628 11873 tablet_bootstrap.cc:439] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:37.029945 11873 tablet_bootstrap.cc:590] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:37.030282 11765 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:37.030323 11765 ts_tablet_manager.cc:1104] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:37.030470 11765 raft_consensus.cc:337] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.030489 11765 raft_consensus.cc:363] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.030496 11765 raft_consensus.cc:702] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:37.030552 11765 consensus_queue.cc:231] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.030622 11765 ts_tablet_manager.cc:1132] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.030817 11768 tablet_bootstrap.cc:439] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:37.030858 11768 ts_tablet_manager.cc:1104] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:37.030989 11768 raft_consensus.cc:337] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.031008 11768 raft_consensus.cc:363] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.031015 11768 raft_consensus.cc:702] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:37.031069 11768 consensus_queue.cc:231] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.031101 11767 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:37.031133 11767 ts_tablet_manager.cc:1104] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:37.031153 11768 ts_tablet_manager.cc:1132] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.031189 11768 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:37.031260 11767 raft_consensus.cc:337] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.031359 11767 raft_consensus.cc:363] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.031368 11767 raft_consensus.cc:702] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:37.031430 11767 consensus_queue.cc:231] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.031489 11768 tablet_bootstrap.cc:590] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:37.031497 11767 ts_tablet_manager.cc:1132] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.031684 11873 tablet_bootstrap.cc:439] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:37.031740 11873 ts_tablet_manager.cc:1104] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:37.031909 11873 raft_consensus.cc:337] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.031930 11873 raft_consensus.cc:363] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.031939 11873 raft_consensus.cc:702] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:37.031998 11873 consensus_queue.cc:231] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.032069 11873 ts_tablet_manager.cc:1132] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.032105 11873 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:37.032382 11873 tablet_bootstrap.cc:590] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:37.033370 11873 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:37.033403 11768 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:37.033419 11873 ts_tablet_manager.cc:1104] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:37.033426 11768 ts_tablet_manager.cc:1104] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:37.033563 11873 raft_consensus.cc:337] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.033566 11768 raft_consensus.cc:337] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.033582 11768 raft_consensus.cc:363] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.033582 11873 raft_consensus.cc:363] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.033586 11768 raft_consensus.cc:702] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:37.033592 11873 raft_consensus.cc:702] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:37.033640 11768 consensus_queue.cc:231] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.033649 11873 consensus_queue.cc:231] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.033731 11768 ts_tablet_manager.cc:1132] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.033716 11873 ts_tablet_manager.cc:1132] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.042038 11907 raft_consensus.cc:469] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:37.042135 11907 raft_consensus.cc:491] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.042253 11907 leader_election.cc:296] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:37.042426 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:37.042471 10953 raft_consensus.cc:2359] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.042613 11137 leader_election.cc:310] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:37.042688 11907 raft_consensus.cc:2653] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:37.042702 11907 raft_consensus.cc:469] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:37.042709 11907 raft_consensus.cc:2905] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.043503 11907 raft_consensus.cc:491] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.043581 11907 leader_election.cc:296] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:37.043732 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:37.043771 10953 raft_consensus.cc:2905] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.045246 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:37.045296 11070 raft_consensus.cc:2359] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.045296 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:37.045315 11080 raft_consensus.cc:2905] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.045837 11080 raft_consensus.cc:2359] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.045994 11135 leader_election.cc:310] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:37.046085 11907 raft_consensus.cc:2653] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:37.046128 11907 raft_consensus.cc:664] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Becoming Leader. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Running, Role: LEADER I1121 10:21:37.046183 11907 consensus_queue.cc:208] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.046725 10707 catalog_manager.cc:4143] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: term changed from 0 to 1, leader changed from to 5457a73f9ecb40ee9889513a8412937b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } } I1121 10:21:37.052563 11889 raft_consensus.cc:469] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:37.052645 11889 raft_consensus.cc:491] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.052772 11889 leader_election.cc:296] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.053004 10955 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "df2330a22c8142ada6d056401d0a064d" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:37.053041 10955 raft_consensus.cc:2359] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.053081 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "df2330a22c8142ada6d056401d0a064d" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:37.053125 10821 raft_consensus.cc:2359] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.053153 11137 leader_election.cc:310] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:37.053208 11889 raft_consensus.cc:2653] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:37.053277 11889 raft_consensus.cc:469] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:37.053289 11889 raft_consensus.cc:2905] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.053853 11889 raft_consensus.cc:491] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.053925 11889 leader_election.cc:296] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.054092 10955 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "df2330a22c8142ada6d056401d0a064d" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:37.054124 10955 raft_consensus.cc:2905] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.054239 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "df2330a22c8142ada6d056401d0a064d" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:37.054270 10821 raft_consensus.cc:2905] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.054791 10821 raft_consensus.cc:2359] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.054967 11135 leader_election.cc:310] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:37.055043 11889 raft_consensus.cc:2653] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:37.055120 11889 raft_consensus.cc:664] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Becoming Leader. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Running, Role: LEADER I1121 10:21:37.055176 11889 consensus_queue.cc:208] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.055706 10707 catalog_manager.cc:4143] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b reported cstate change: term changed from 0 to 1, leader changed from to 5457a73f9ecb40ee9889513a8412937b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } } I1121 10:21:37.069321 10953 raft_consensus.cc:2359] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.069423 10955 raft_consensus.cc:2359] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.251453 11949 raft_consensus.cc:469] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:37.251549 11949 raft_consensus.cc:491] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.251642 11949 leader_election.cc:296] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.251899 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "204d260bb58b494091ae91cb57a88ca8" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:37.251899 10955 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "204d260bb58b494091ae91cb57a88ca8" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:37.251946 10955 raft_consensus.cc:2359] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:37.251966 10821 raft_consensus.cc:2359] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:37.252077 11013 leader_election.cc:310] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:37.252158 11949 raft_consensus.cc:2653] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:37.252182 11949 raft_consensus.cc:469] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:37.252207 11949 raft_consensus.cc:2905] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.252907 11949 raft_consensus.cc:491] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.252984 11949 leader_election.cc:296] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.253151 10955 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "204d260bb58b494091ae91cb57a88ca8" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:37.253177 10821 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "204d260bb58b494091ae91cb57a88ca8" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:37.253183 10955 raft_consensus.cc:2905] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.253197 10821 raft_consensus.cc:2905] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.253701 10821 raft_consensus.cc:2359] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:37.253737 10955 raft_consensus.cc:2359] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:37.253937 11011 leader_election.cc:310] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:37.254012 11949 raft_consensus.cc:2653] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:37.254061 11949 raft_consensus.cc:664] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:37.254128 11949 consensus_queue.cc:208] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.254726 10707 catalog_manager.cc:4143] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } } W1121 10:21:37.363101 11971 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_4' I1121 10:21:37.390713 10955 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:37.390733 10953 raft_consensus.cc:1181] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:37.390714 10821 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:37.390728 10816 raft_consensus.cc:1181] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:37.390980 11889 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.390987 11948 consensus_queue.cc:984] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.391137 11889 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.391336 11948 consensus_queue.cc:984] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.391422 10955 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.392854 11987 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.392912 11080 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.394511 11889 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 22.1038 ms time per row: 0.0110519 ms I1121 10:21:37.401697 11993 mvcc.cc:203] Tried to move safe_time back from 6448462632515194880 to 6448462631101501440. Current Snapshot: MvccSnapshot[committed={T|T < 6448462632515194880 or (T in {6448462632515194880})}] I1121 10:21:37.405522 10707 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_5" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:37.407034 11061 tablet_service.cc:1023] Processing CreateTablet for tablet 1fa5ac03774f4944815d262fcdffa2f6 (table=rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.407050 10807 tablet_service.cc:1023] Processing CreateTablet for tablet 1fa5ac03774f4944815d262fcdffa2f6 (table=rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.407109 11185 tablet_service.cc:1023] Processing CreateTablet for tablet 1fa5ac03774f4944815d262fcdffa2f6 (table=rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.407151 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 1fa5ac03774f4944815d262fcdffa2f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.407155 11061 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 1fa5ac03774f4944815d262fcdffa2f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.407194 11185 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 1fa5ac03774f4944815d262fcdffa2f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.407244 10935 tablet_service.cc:1023] Processing CreateTablet for tablet fa85115a8bbd45dbb7ce1f18b923022a (table=rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.407263 10804 tablet_service.cc:1023] Processing CreateTablet for tablet 0e5ec5084c6a4d6089c2cb198367d342 (table=rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.407323 10804 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0e5ec5084c6a4d6089c2cb198367d342. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.407323 10935 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet fa85115a8bbd45dbb7ce1f18b923022a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.407407 10805 tablet_service.cc:1023] Processing CreateTablet for tablet fa85115a8bbd45dbb7ce1f18b923022a (table=rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.407465 10805 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet fa85115a8bbd45dbb7ce1f18b923022a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.408228 11768 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:37.408373 11176 tablet_service.cc:1023] Processing CreateTablet for tablet 0e5ec5084c6a4d6089c2cb198367d342 (table=rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.408380 11059 tablet_service.cc:1023] Processing CreateTablet for tablet 0e5ec5084c6a4d6089c2cb198367d342 (table=rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.408442 11059 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0e5ec5084c6a4d6089c2cb198367d342. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.408442 11176 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0e5ec5084c6a4d6089c2cb198367d342. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.408589 11060 tablet_service.cc:1023] Processing CreateTablet for tablet fa85115a8bbd45dbb7ce1f18b923022a (table=rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.408653 11060 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet fa85115a8bbd45dbb7ce1f18b923022a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.408713 11873 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:37.408732 11768 tablet_bootstrap.cc:590] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:37.409278 11768 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:37.409303 11768 ts_tablet_manager.cc:1104] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:37.409406 11768 raft_consensus.cc:337] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:37.409415 11768 raft_consensus.cc:363] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.409420 11768 raft_consensus.cc:702] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:37.409479 11768 consensus_queue.cc:231] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:37.409477 11765 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:37.409545 11768 ts_tablet_manager.cc:1132] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.409775 11765 tablet_bootstrap.cc:590] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:37.410333 11765 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:37.410372 11765 ts_tablet_manager.cc:1104] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:37.410513 11765 raft_consensus.cc:337] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.410534 11765 raft_consensus.cc:363] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.410545 11765 raft_consensus.cc:702] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:37.410609 11765 consensus_queue.cc:231] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.410687 11765 ts_tablet_manager.cc:1132] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.410723 11765 tablet_bootstrap.cc:439] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:37.416622 11873 tablet_bootstrap.cc:590] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:37.417239 11873 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:37.417280 11873 ts_tablet_manager.cc:1104] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s I1121 10:21:37.417414 11873 raft_consensus.cc:337] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.417428 11873 raft_consensus.cc:363] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.417434 11873 raft_consensus.cc:702] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:37.417475 11873 consensus_queue.cc:231] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.417539 11873 ts_tablet_manager.cc:1132] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.417582 11873 tablet_bootstrap.cc:439] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:37.417917 11873 tablet_bootstrap.cc:590] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:37.418347 11818 mvcc.cc:203] Tried to move safe_time back from 6448462632515194880 to 6448462631101501440. Current Snapshot: MvccSnapshot[committed={T|T < 6448462632515194880 or (T in {6448462632515194880})}] I1121 10:21:37.418613 11873 tablet_bootstrap.cc:439] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:37.418653 11873 ts_tablet_manager.cc:1104] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:37.418792 11873 raft_consensus.cc:337] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.418812 11873 raft_consensus.cc:363] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.418817 11873 raft_consensus.cc:702] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:37.418885 11873 consensus_queue.cc:231] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.418973 11873 ts_tablet_manager.cc:1132] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.419013 11873 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:37.419304 11873 tablet_bootstrap.cc:590] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:37.419780 11873 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:37.419816 11873 ts_tablet_manager.cc:1104] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:37.419977 11873 raft_consensus.cc:337] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:37.419996 11873 raft_consensus.cc:363] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.420001 11873 raft_consensus.cc:702] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:37.420051 11873 consensus_queue.cc:231] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:37.420114 11873 ts_tablet_manager.cc:1132] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.425438 11767 tablet_bootstrap.cc:439] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:37.425762 11767 tablet_bootstrap.cc:590] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:37.426194 11767 tablet_bootstrap.cc:439] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:37.426229 11767 ts_tablet_manager.cc:1104] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:37.426342 11767 raft_consensus.cc:337] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.426358 11767 raft_consensus.cc:363] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.426363 11767 raft_consensus.cc:702] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:37.426414 11767 consensus_queue.cc:231] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.426527 11767 ts_tablet_manager.cc:1132] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.426563 11767 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:37.426854 11767 tablet_bootstrap.cc:590] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:37.427356 11767 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:37.427389 11767 ts_tablet_manager.cc:1104] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:37.427505 11767 raft_consensus.cc:337] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.427523 11767 raft_consensus.cc:363] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.427532 11767 raft_consensus.cc:702] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:37.427589 11767 consensus_queue.cc:231] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.427667 11767 ts_tablet_manager.cc:1132] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.427708 11767 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:37.428020 11767 tablet_bootstrap.cc:590] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:37.428728 11767 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:37.428791 11767 ts_tablet_manager.cc:1104] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:37.428999 11767 raft_consensus.cc:337] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:37.429023 11767 raft_consensus.cc:363] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.429029 11767 raft_consensus.cc:702] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:37.429095 11767 consensus_queue.cc:231] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:37.429188 11767 ts_tablet_manager.cc:1132] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.436638 11765 tablet_bootstrap.cc:590] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:37.437160 11765 tablet_bootstrap.cc:439] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:37.437196 11765 ts_tablet_manager.cc:1104] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.001s I1121 10:21:37.437376 11765 raft_consensus.cc:337] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.437407 11765 raft_consensus.cc:363] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.437413 11765 raft_consensus.cc:702] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:37.437474 11765 consensus_queue.cc:231] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.437588 11765 ts_tablet_manager.cc:1132] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.495713 11986 raft_consensus.cc:469] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:37.495863 11986 raft_consensus.cc:491] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.495939 11986 leader_election.cc:296] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:37.496183 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:37.496220 11080 raft_consensus.cc:2359] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.496284 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:37.496323 10816 raft_consensus.cc:2359] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.496371 11135 leader_election.cc:310] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:37.496484 11986 raft_consensus.cc:2653] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:37.496502 11986 raft_consensus.cc:469] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:37.496547 11986 raft_consensus.cc:2905] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.497186 11986 raft_consensus.cc:491] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.497259 11986 leader_election.cc:296] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Requested vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:37.497488 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:37.497493 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:37.497520 10816 raft_consensus.cc:2905] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.497521 11080 raft_consensus.cc:2905] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.498061 11080 raft_consensus.cc:2359] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.498077 10816 raft_consensus.cc:2359] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.498289 11135 leader_election.cc:310] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:37.498374 11986 raft_consensus.cc:2653] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:37.498420 11986 raft_consensus.cc:664] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Becoming Leader. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Running, Role: LEADER I1121 10:21:37.498486 11986 consensus_queue.cc:208] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:37.499140 10707 catalog_manager.cc:4143] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: term changed from 0 to 1, leader changed from to 5457a73f9ecb40ee9889513a8412937b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } } I1121 10:21:37.510067 11986 raft_consensus.cc:469] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:37.510154 11986 raft_consensus.cc:491] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.510349 11986 leader_election.cc:296] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.510629 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0e5ec5084c6a4d6089c2cb198367d342" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:37.510629 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0e5ec5084c6a4d6089c2cb198367d342" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:37.510676 10816 raft_consensus.cc:2359] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.510676 11080 raft_consensus.cc:2359] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.510804 11135 leader_election.cc:310] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:37.510898 11986 raft_consensus.cc:2653] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:37.510921 11986 raft_consensus.cc:469] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:37.510932 11986 raft_consensus.cc:2905] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.511495 11986 raft_consensus.cc:491] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.511564 11986 leader_election.cc:296] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Requested vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.511781 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0e5ec5084c6a4d6089c2cb198367d342" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:37.511809 11080 raft_consensus.cc:2905] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.511765 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0e5ec5084c6a4d6089c2cb198367d342" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:37.511831 10816 raft_consensus.cc:2905] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.512332 11080 raft_consensus.cc:2359] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.512359 10816 raft_consensus.cc:2359] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.512480 11135 leader_election.cc:310] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:37.512593 11986 raft_consensus.cc:2653] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:37.512630 11986 raft_consensus.cc:664] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Becoming Leader. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Running, Role: LEADER I1121 10:21:37.512696 11986 consensus_queue.cc:208] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.513312 10707 catalog_manager.cc:4143] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: term changed from 0 to 1, leader changed from to 5457a73f9ecb40ee9889513a8412937b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } } I1121 10:21:37.563977 11949 raft_consensus.cc:469] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:37.564092 11949 raft_consensus.cc:491] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:37.564258 11949 leader_election.cc:296] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:37.564404 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:37.564450 10816 raft_consensus.cc:2359] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:37.564456 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:37.564487 10953 raft_consensus.cc:2359] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:37.564630 11011 leader_election.cc:310] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:37.564731 11949 raft_consensus.cc:2653] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:37.564747 11949 raft_consensus.cc:469] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:37.564755 11949 raft_consensus.cc:2905] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.565409 11949 raft_consensus.cc:491] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:37.565482 11949 leader_election.cc:296] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:37.565654 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:37.565685 10816 raft_consensus.cc:2905] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.565716 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:37.565733 10953 raft_consensus.cc:2905] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.566186 10816 raft_consensus.cc:2359] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:37.566242 10953 raft_consensus.cc:2359] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:37.566390 11013 leader_election.cc:310] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:37.566565 11949 raft_consensus.cc:2653] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:37.566615 11949 raft_consensus.cc:664] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:37.566689 11949 consensus_queue.cc:208] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:37.567417 10707 catalog_manager.cc:4143] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } } W1121 10:21:37.644241 12010 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_5' I1121 10:21:37.670661 11080 raft_consensus.cc:1181] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.670673 10816 raft_consensus.cc:1181] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.670909 11986 consensus_queue.cc:984] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.670990 11907 consensus_queue.cc:984] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.671416 10816 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:37.671607 11789 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.671998 10953 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:37.672188 11789 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.672922 10821 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.673127 11070 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:37.673350 11987 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.673456 11987 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:37.675591 11812 mvcc.cc:203] Tried to move safe_time back from 6448462633663979520 to 6448462632953946112. Current Snapshot: MvccSnapshot[committed={T|T < 6448462633663979520}] Generator report time total : 20.7324 ms time per row: 0.0103662 ms I1121 10:21:37.681507 11912 mvcc.cc:203] Tried to move safe_time back from 6448462633660841984 to 6448462633233575936. Current Snapshot: MvccSnapshot[committed={T|T < 6448462633660841984 or (T in {6448462633660841984})}] I1121 10:21:37.684324 10707 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:37.685686 10935 tablet_service.cc:1023] Processing CreateTablet for tablet 172c695aadee4de6ad0a5a6508b290f6 (table=rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.685782 10805 tablet_service.cc:1023] Processing CreateTablet for tablet 25b55ff241d8442897ea97523a2cd6fd (table=rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.685813 10935 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 172c695aadee4de6ad0a5a6508b290f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.685842 10807 tablet_service.cc:1023] Processing CreateTablet for tablet 172c695aadee4de6ad0a5a6508b290f6 (table=rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.685863 10805 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 25b55ff241d8442897ea97523a2cd6fd. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.685930 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 172c695aadee4de6ad0a5a6508b290f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.685956 11176 tablet_service.cc:1023] Processing CreateTablet for tablet 172c695aadee4de6ad0a5a6508b290f6 (table=rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.686039 11176 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 172c695aadee4de6ad0a5a6508b290f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.687014 11767 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:37.687041 10936 tablet_service.cc:1023] Processing CreateTablet for tablet 25b55ff241d8442897ea97523a2cd6fd (table=rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.687074 11060 tablet_service.cc:1023] Processing CreateTablet for tablet 45f0a219c25243fbade25321fcc1067a (table=rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.687124 10936 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 25b55ff241d8442897ea97523a2cd6fd. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.687168 11060 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 45f0a219c25243fbade25321fcc1067a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.687227 11185 tablet_service.cc:1023] Processing CreateTablet for tablet 45f0a219c25243fbade25321fcc1067a (table=rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.687311 11185 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 45f0a219c25243fbade25321fcc1067a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.687419 11767 tablet_bootstrap.cc:590] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:37.688174 11768 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:37.688308 11180 tablet_service.cc:1023] Processing CreateTablet for tablet 25b55ff241d8442897ea97523a2cd6fd (table=rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.688310 11873 tablet_bootstrap.cc:439] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:37.688380 11180 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 25b55ff241d8442897ea97523a2cd6fd. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.688427 11768 tablet_bootstrap.cc:590] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:37.685686 10937 tablet_service.cc:1023] Processing CreateTablet for tablet 45f0a219c25243fbade25321fcc1067a (table=rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:37.688575 11765 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:37.688642 10937 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 45f0a219c25243fbade25321fcc1067a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:37.688776 11873 tablet_bootstrap.cc:590] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:37.688897 11765 tablet_bootstrap.cc:590] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:37.689692 11765 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:37.689728 11765 ts_tablet_manager.cc:1104] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:37.689865 11765 raft_consensus.cc:337] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.689893 11765 raft_consensus.cc:363] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.689901 11765 raft_consensus.cc:702] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:37.689970 11765 consensus_queue.cc:231] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.690065 11765 ts_tablet_manager.cc:1132] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.690106 11765 tablet_bootstrap.cc:439] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:37.690110 11768 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:37.690138 11768 ts_tablet_manager.cc:1104] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:37.690268 11768 raft_consensus.cc:337] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.690280 11768 raft_consensus.cc:363] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.690287 11768 raft_consensus.cc:702] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:37.690337 11768 consensus_queue.cc:231] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.690412 11768 ts_tablet_manager.cc:1132] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.690443 11768 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:37.690448 11765 tablet_bootstrap.cc:590] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:37.690766 11768 tablet_bootstrap.cc:590] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:37.691267 11873 tablet_bootstrap.cc:439] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:37.691267 11768 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:37.691306 11873 ts_tablet_manager.cc:1104] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:37.691306 11768 ts_tablet_manager.cc:1104] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:37.691449 11768 raft_consensus.cc:337] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.691462 11873 raft_consensus.cc:337] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.691524 11873 raft_consensus.cc:363] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.691524 11768 raft_consensus.cc:363] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.691534 11768 raft_consensus.cc:702] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:37.691534 11873 raft_consensus.cc:702] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:37.691592 11873 consensus_queue.cc:231] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.691592 11768 consensus_queue.cc:231] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.691670 11873 ts_tablet_manager.cc:1132] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.691671 11768 ts_tablet_manager.cc:1132] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.691723 11768 tablet_bootstrap.cc:439] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:37.691900 11767 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:37.691936 11767 ts_tablet_manager.cc:1104] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s I1121 10:21:37.692029 11768 tablet_bootstrap.cc:590] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:37.692072 11767 raft_consensus.cc:337] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.692088 11767 raft_consensus.cc:363] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.692096 11767 raft_consensus.cc:702] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:37.692160 11767 consensus_queue.cc:231] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.692247 11767 ts_tablet_manager.cc:1132] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.692289 11767 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:37.692600 11767 tablet_bootstrap.cc:590] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:37.693053 11768 tablet_bootstrap.cc:439] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:37.693053 11767 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:37.693089 11768 ts_tablet_manager.cc:1104] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:37.693089 11767 ts_tablet_manager.cc:1104] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:37.693213 11768 raft_consensus.cc:337] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.693217 11767 raft_consensus.cc:337] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.693238 11767 raft_consensus.cc:363] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.693238 11768 raft_consensus.cc:363] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.693245 11768 raft_consensus.cc:702] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:37.693245 11767 raft_consensus.cc:702] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:37.693300 11768 consensus_queue.cc:231] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.693300 11767 consensus_queue.cc:231] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.693379 11767 ts_tablet_manager.cc:1132] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.693382 11768 ts_tablet_manager.cc:1132] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.693558 11765 tablet_bootstrap.cc:439] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:37.693593 11765 ts_tablet_manager.cc:1104] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I1121 10:21:37.693728 11765 raft_consensus.cc:337] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.693742 11765 raft_consensus.cc:363] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.693747 11765 raft_consensus.cc:702] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:37.693804 11765 consensus_queue.cc:231] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.693856 11907 raft_consensus.cc:469] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:37.693874 11765 ts_tablet_manager.cc:1132] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.693907 11765 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:37.693922 11907 raft_consensus.cc:491] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.694002 11907 leader_election.cc:296] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.694159 11765 tablet_bootstrap.cc:590] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:37.694176 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "172c695aadee4de6ad0a5a6508b290f6" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:37.694217 10953 raft_consensus.cc:2359] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.694356 11137 leader_election.cc:310] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:37.694429 11907 raft_consensus.cc:2653] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:37.694450 11907 raft_consensus.cc:469] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:37.694458 11907 raft_consensus.cc:2905] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.694525 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "172c695aadee4de6ad0a5a6508b290f6" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:37.694561 10816 raft_consensus.cc:2359] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:37.694859 11765 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:37.694895 11765 ts_tablet_manager.cc:1104] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:37.695014 11765 raft_consensus.cc:337] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.695034 11765 raft_consensus.cc:363] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:37.695040 11765 raft_consensus.cc:702] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:37.695096 11765 consensus_queue.cc:231] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.695127 11907 raft_consensus.cc:491] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.695168 11765 ts_tablet_manager.cc:1132] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:37.695188 11907 leader_election.cc:296] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.695375 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "172c695aadee4de6ad0a5a6508b290f6" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:37.695405 10953 raft_consensus.cc:2905] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.695478 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "172c695aadee4de6ad0a5a6508b290f6" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:37.695509 10816 raft_consensus.cc:2905] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.696003 10953 raft_consensus.cc:2359] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.696111 10816 raft_consensus.cc:2359] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:37.696162 11137 leader_election.cc:310] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:37.696254 11907 raft_consensus.cc:2653] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:37.696307 11907 raft_consensus.cc:664] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Becoming Leader. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Running, Role: LEADER I1121 10:21:37.696380 11907 consensus_queue.cc:208] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.696907 10707 catalog_manager.cc:4143] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: term changed from 0 to 1, leader changed from to 5457a73f9ecb40ee9889513a8412937b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } } I1121 10:21:37.866287 11822 raft_consensus.cc:469] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:37.866411 11822 raft_consensus.cc:491] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.866515 11822 leader_election.cc:296] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:37.866734 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "45f0a219c25243fbade25321fcc1067a" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:37.866783 11204 raft_consensus.cc:2359] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:37.866798 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "45f0a219c25243fbade25321fcc1067a" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:37.866834 11080 raft_consensus.cc:2359] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:37.866925 10890 leader_election.cc:310] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:37.867002 11822 raft_consensus.cc:2653] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:37.867022 11822 raft_consensus.cc:469] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:37.867030 11822 raft_consensus.cc:2905] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.867767 11822 raft_consensus.cc:491] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.867846 11822 leader_election.cc:296] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Requested vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:37.868028 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "45f0a219c25243fbade25321fcc1067a" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:37.868063 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "45f0a219c25243fbade25321fcc1067a" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:37.868070 11204 raft_consensus.cc:2905] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.868086 11080 raft_consensus.cc:2905] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.868868 11204 raft_consensus.cc:2359] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:37.868868 11080 raft_consensus.cc:2359] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:37.869096 10887 leader_election.cc:310] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:37.869185 11822 raft_consensus.cc:2653] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:37.869254 11822 raft_consensus.cc:664] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:37.869321 11822 consensus_queue.cc:208] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:37.869999 10707 catalog_manager.cc:4143] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 0 to 1, leader changed from to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } } I1121 10:21:37.985430 11822 raft_consensus.cc:469] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:37.985522 11822 raft_consensus.cc:491] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.985630 11822 leader_election.cc:296] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.985882 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:37.985898 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:37.985926 10816 raft_consensus.cc:2359] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:37.985960 11204 raft_consensus.cc:2359] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:37.986064 10887 leader_election.cc:310] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:37.986148 11822 raft_consensus.cc:2653] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:37.986174 11822 raft_consensus.cc:469] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:37.986183 11822 raft_consensus.cc:2905] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.986843 11822 raft_consensus.cc:491] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.986915 11822 leader_election.cc:296] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Requested vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:37.987087 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:37.987123 11204 raft_consensus.cc:2905] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.987267 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:37.987298 10816 raft_consensus.cc:2905] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:37.987823 11204 raft_consensus.cc:2359] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:37.988004 10816 raft_consensus.cc:2359] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:37.988044 10890 leader_election.cc:310] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:37.988108 11822 raft_consensus.cc:2653] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:37.988162 11822 raft_consensus.cc:664] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:37.988226 11822 consensus_queue.cc:208] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:37.988955 10707 catalog_manager.cc:4143] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 0 to 1, leader changed from to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } } W1121 10:21:38.100036 12043 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_6' I1121 10:21:38.127192 10816 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:38.127233 11080 raft_consensus.cc:1181] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:38.127399 11204 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:38.127473 11931 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.127787 11204 raft_consensus.cc:1181] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:38.127851 11822 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.127878 11931 consensus_queue.cc:984] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.128084 11822 consensus_queue.cc:984] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.129328 10953 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:38.129549 11907 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.130393 10815 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:38.130612 11986 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 22.8142 ms time per row: 0.0114071 ms I1121 10:21:38.142776 10707 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:38.144367 10937 tablet_service.cc:1023] Processing CreateTablet for tablet 3e0bc1c59a1b4e2c9e761a6de8085226 (table=rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.144417 10807 tablet_service.cc:1023] Processing CreateTablet for tablet 3e0bc1c59a1b4e2c9e761a6de8085226 (table=rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.144426 11180 tablet_service.cc:1023] Processing CreateTablet for tablet 0ec67ec228044beda81b53a3e842af49 (table=rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.144459 11060 tablet_service.cc:1023] Processing CreateTablet for tablet 3e0bc1c59a1b4e2c9e761a6de8085226 (table=rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.144491 10937 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 3e0bc1c59a1b4e2c9e761a6de8085226. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.144592 10936 tablet_service.cc:1023] Processing CreateTablet for tablet 0ec67ec228044beda81b53a3e842af49 (table=rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.144596 11059 tablet_service.cc:1023] Processing CreateTablet for tablet 0ec67ec228044beda81b53a3e842af49 (table=rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.144619 10805 tablet_service.cc:1023] Processing CreateTablet for tablet a40d6cf45df24788adc85f238c9bcc7e (table=rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.144657 10936 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0ec67ec228044beda81b53a3e842af49. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.144673 11059 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0ec67ec228044beda81b53a3e842af49. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.144701 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 3e0bc1c59a1b4e2c9e761a6de8085226. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.144685 11180 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0ec67ec228044beda81b53a3e842af49. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.145892 11765 tablet_bootstrap.cc:439] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:38.145892 11768 tablet_bootstrap.cc:439] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:38.145896 11767 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:38.145915 11061 tablet_service.cc:1023] Processing CreateTablet for tablet a40d6cf45df24788adc85f238c9bcc7e (table=rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.145984 11061 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet a40d6cf45df24788adc85f238c9bcc7e. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.146189 11767 tablet_bootstrap.cc:590] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:38.146211 11768 tablet_bootstrap.cc:590] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:38.146222 11765 tablet_bootstrap.cc:590] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:38.146674 10805 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet a40d6cf45df24788adc85f238c9bcc7e. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.146788 11180 tablet_service.cc:1023] Processing CreateTablet for tablet a40d6cf45df24788adc85f238c9bcc7e (table=rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.146859 11180 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet a40d6cf45df24788adc85f238c9bcc7e. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.147019 11873 tablet_bootstrap.cc:439] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:38.147331 11873 tablet_bootstrap.cc:590] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:38.147668 11060 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 3e0bc1c59a1b4e2c9e761a6de8085226. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.148000 11767 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:38.148046 11767 ts_tablet_manager.cc:1104] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:38.148185 11767 raft_consensus.cc:337] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.148206 11767 raft_consensus.cc:363] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.148214 11767 raft_consensus.cc:702] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:38.148272 11767 consensus_queue.cc:231] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.148366 11767 ts_tablet_manager.cc:1132] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.148406 11767 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:38.148752 11767 tablet_bootstrap.cc:590] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:38.149080 11765 tablet_bootstrap.cc:439] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:38.149130 11765 ts_tablet_manager.cc:1104] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s I1121 10:21:38.149287 11765 raft_consensus.cc:337] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.149313 11765 raft_consensus.cc:363] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.149322 11765 raft_consensus.cc:702] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:38.149377 11765 consensus_queue.cc:231] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.149421 11873 tablet_bootstrap.cc:439] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:38.149457 11765 ts_tablet_manager.cc:1132] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.149466 11873 ts_tablet_manager.cc:1104] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:38.149493 11765 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:38.149606 11873 raft_consensus.cc:337] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.149634 11873 raft_consensus.cc:363] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.149643 11873 raft_consensus.cc:702] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:38.149724 11873 consensus_queue.cc:231] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.149811 11873 ts_tablet_manager.cc:1132] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.149852 11873 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:38.149884 11765 tablet_bootstrap.cc:590] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:38.149945 11767 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:38.149988 11767 ts_tablet_manager.cc:1104] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I1121 10:21:38.150118 11767 raft_consensus.cc:337] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.150143 11767 raft_consensus.cc:363] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.150151 11767 raft_consensus.cc:702] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:38.150156 11873 tablet_bootstrap.cc:590] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:38.150223 11767 consensus_queue.cc:231] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.150300 11767 ts_tablet_manager.cc:1132] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.150779 11873 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:38.150813 11873 ts_tablet_manager.cc:1104] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:38.150933 11873 raft_consensus.cc:337] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.150959 11873 raft_consensus.cc:363] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.150967 11873 raft_consensus.cc:702] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:38.150993 11768 tablet_bootstrap.cc:439] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:38.151029 11768 ts_tablet_manager.cc:1104] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.000s I1121 10:21:38.151032 11873 consensus_queue.cc:231] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.151108 11873 ts_tablet_manager.cc:1132] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.151146 11873 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:38.151190 11768 raft_consensus.cc:337] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.151218 11768 raft_consensus.cc:363] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.151229 11768 raft_consensus.cc:702] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:38.151294 11768 consensus_queue.cc:231] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.151376 11768 ts_tablet_manager.cc:1132] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.151427 11768 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:38.151448 11873 tablet_bootstrap.cc:590] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:38.151767 11768 tablet_bootstrap.cc:590] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:38.152297 11768 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:38.152309 11765 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:38.152333 11768 ts_tablet_manager.cc:1104] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:38.152346 11765 ts_tablet_manager.cc:1104] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s I1121 10:21:38.152359 11873 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:38.152384 11873 ts_tablet_manager.cc:1104] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:38.152492 11768 raft_consensus.cc:337] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.152504 11873 raft_consensus.cc:337] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.152494 11765 raft_consensus.cc:337] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.152508 11768 raft_consensus.cc:363] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.152551 11873 raft_consensus.cc:363] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.152551 11765 raft_consensus.cc:363] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.152559 11768 raft_consensus.cc:702] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:38.152560 11765 raft_consensus.cc:702] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:38.152560 11873 raft_consensus.cc:702] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:38.152616 11765 consensus_queue.cc:231] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.152616 11873 consensus_queue.cc:231] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.152626 11768 consensus_queue.cc:231] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.152673 11873 ts_tablet_manager.cc:1132] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.152673 11765 ts_tablet_manager.cc:1132] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.152747 11768 ts_tablet_manager.cc:1132] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.172296 11949 raft_consensus.cc:469] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:38.172371 11949 raft_consensus.cc:491] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.172444 11949 leader_election.cc:296] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:38.172648 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "a40d6cf45df24788adc85f238c9bcc7e" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:38.172672 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "a40d6cf45df24788adc85f238c9bcc7e" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:38.172704 10815 raft_consensus.cc:2359] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:38.172755 11204 raft_consensus.cc:2359] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:38.172850 11011 leader_election.cc:310] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:38.172935 11949 raft_consensus.cc:2653] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:38.172958 11949 raft_consensus.cc:469] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:38.172973 11949 raft_consensus.cc:2905] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.173622 11949 raft_consensus.cc:491] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.173714 11949 leader_election.cc:296] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:38.173940 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "a40d6cf45df24788adc85f238c9bcc7e" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:38.173961 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "a40d6cf45df24788adc85f238c9bcc7e" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:38.173976 11204 raft_consensus.cc:2905] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.173981 10815 raft_consensus.cc:2905] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.174690 11204 raft_consensus.cc:2359] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:38.174690 10815 raft_consensus.cc:2359] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:38.174957 11011 leader_election.cc:310] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:38.175052 11949 raft_consensus.cc:2653] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:38.175099 11949 raft_consensus.cc:664] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:38.175160 11949 consensus_queue.cc:208] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.175698 10707 catalog_manager.cc:4143] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } } I1121 10:21:38.296066 12067 raft_consensus.cc:469] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:38.296173 12067 raft_consensus.cc:491] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.296340 12067 leader_election.cc:296] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:38.296581 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0ec67ec228044beda81b53a3e842af49" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:38.296643 11080 raft_consensus.cc:2359] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:38.296644 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0ec67ec228044beda81b53a3e842af49" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:38.296690 11204 raft_consensus.cc:2359] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:38.296854 10890 leader_election.cc:310] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:38.296945 12067 raft_consensus.cc:2653] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:38.296979 12067 raft_consensus.cc:469] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:38.296991 12067 raft_consensus.cc:2905] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.297669 12067 raft_consensus.cc:491] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.297811 12067 leader_election.cc:296] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Requested vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:38.297886 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0ec67ec228044beda81b53a3e842af49" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:38.297904 11204 raft_consensus.cc:2905] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.297956 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0ec67ec228044beda81b53a3e842af49" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:38.297993 11080 raft_consensus.cc:2905] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.298534 11080 raft_consensus.cc:2359] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:38.298633 11204 raft_consensus.cc:2359] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:38.298746 10887 leader_election.cc:310] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:38.298821 12067 raft_consensus.cc:2653] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:38.298858 12067 raft_consensus.cc:664] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:38.298916 12067 consensus_queue.cc:208] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.299518 10707 catalog_manager.cc:4143] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 0 to 1, leader changed from to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } } I1121 10:21:38.457976 11991 raft_consensus.cc:469] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:38.458091 11991 raft_consensus.cc:491] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.458190 11991 leader_election.cc:296] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:38.458473 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "3e0bc1c59a1b4e2c9e761a6de8085226" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:38.458501 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "3e0bc1c59a1b4e2c9e761a6de8085226" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:38.458518 11080 raft_consensus.cc:2359] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 0. I1121 10:21:38.458534 10953 raft_consensus.cc:2359] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 0. I1121 10:21:38.458655 10762 leader_election.cc:310] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:38.458739 11991 raft_consensus.cc:2653] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:38.458762 11991 raft_consensus.cc:469] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:38.458770 11991 raft_consensus.cc:2905] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.459514 11991 raft_consensus.cc:491] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.459583 11991 leader_election.cc:296] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:38.459797 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "3e0bc1c59a1b4e2c9e761a6de8085226" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:38.459805 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "3e0bc1c59a1b4e2c9e761a6de8085226" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:38.459830 11080 raft_consensus.cc:2905] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.459830 10953 raft_consensus.cc:2905] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.460578 10953 raft_consensus.cc:2359] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 1. I1121 10:21:38.460578 11080 raft_consensus.cc:2359] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 1. I1121 10:21:38.460839 10762 leader_election.cc:310] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:38.460902 11991 raft_consensus.cc:2653] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:38.460995 11991 raft_consensus.cc:664] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Becoming Leader. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Running, Role: LEADER I1121 10:21:38.461069 11991 consensus_queue.cc:208] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.461830 10707 catalog_manager.cc:4143] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: term changed from 0 to 1, leader changed from to cd693d36f0014dc3a7c3f26196124794 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } } W1121 10:21:38.556911 12081 flags.cc:404] Enabled unsafe flag: --never_fsync=true I1121 10:21:38.566515 11949 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.567932 11948 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Using table 'rebalance_test_table_7' I1121 10:21:38.585911 11080 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:38.585911 10953 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:38.586131 11204 raft_consensus.cc:1181] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:38.586148 12079 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.586350 11080 raft_consensus.cc:1181] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:38.586625 12059 consensus_queue.cc:984] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.587496 12079 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.588393 12058 consensus_queue.cc:984] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 22.3222 ms time per row: 0.0111611 ms I1121 10:21:38.599470 10707 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_8" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:38.600872 10937 tablet_service.cc:1023] Processing CreateTablet for tablet 40e235c631e14bf8b4fe83e2953d8c87 (table=rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.600927 11180 tablet_service.cc:1023] Processing CreateTablet for tablet 114dbe156d634fca957db3164f8afabb (table=rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.600931 11060 tablet_service.cc:1023] Processing CreateTablet for tablet 40e235c631e14bf8b4fe83e2953d8c87 (table=rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.600977 10936 tablet_service.cc:1023] Processing CreateTablet for tablet 114dbe156d634fca957db3164f8afabb (table=rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.600998 10937 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 40e235c631e14bf8b4fe83e2953d8c87. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.601009 11180 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 114dbe156d634fca957db3164f8afabb. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.601023 11060 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 40e235c631e14bf8b4fe83e2953d8c87. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.601050 10936 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 114dbe156d634fca957db3164f8afabb. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.602277 11061 tablet_service.cc:1023] Processing CreateTablet for tablet 4ef99b04fd634cd599c435ad3383c4d5 (table=rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.602296 10935 tablet_service.cc:1023] Processing CreateTablet for tablet 4ef99b04fd634cd599c435ad3383c4d5 (table=rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.602306 11185 tablet_service.cc:1023] Processing CreateTablet for tablet 4ef99b04fd634cd599c435ad3383c4d5 (table=rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.602357 11061 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ef99b04fd634cd599c435ad3383c4d5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.602357 10935 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ef99b04fd634cd599c435ad3383c4d5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.602376 11185 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ef99b04fd634cd599c435ad3383c4d5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.603040 11765 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:38.603332 11765 tablet_bootstrap.cc:590] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:38.603330 11873 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:38.603384 11768 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:38.603660 11768 tablet_bootstrap.cc:590] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:38.603660 11873 tablet_bootstrap.cc:590] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:38.604157 10805 tablet_service.cc:1023] Processing CreateTablet for tablet 40e235c631e14bf8b4fe83e2953d8c87 (table=rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.604260 10805 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 40e235c631e14bf8b4fe83e2953d8c87. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.604600 10807 tablet_service.cc:1023] Processing CreateTablet for tablet 114dbe156d634fca957db3164f8afabb (table=rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.604658 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 114dbe156d634fca957db3164f8afabb. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.605530 11767 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:38.605837 11767 tablet_bootstrap.cc:590] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:38.606531 11765 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:38.606575 11765 ts_tablet_manager.cc:1104] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s I1121 10:21:38.606652 11768 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:38.606693 11768 ts_tablet_manager.cc:1104] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:38.606750 11765 raft_consensus.cc:337] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.606776 11765 raft_consensus.cc:363] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.606784 11765 raft_consensus.cc:702] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:38.606835 11768 raft_consensus.cc:337] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.606840 11765 consensus_queue.cc:231] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.606858 11768 raft_consensus.cc:363] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.606864 11768 raft_consensus.cc:702] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:38.606900 11873 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:38.606916 11765 ts_tablet_manager.cc:1132] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.606920 11768 consensus_queue.cc:231] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.606935 11873 ts_tablet_manager.cc:1104] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s I1121 10:21:38.606950 11765 tablet_bootstrap.cc:439] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:38.607000 11768 ts_tablet_manager.cc:1132] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.607033 11768 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:38.607070 11873 raft_consensus.cc:337] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.607084 11873 raft_consensus.cc:363] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.607092 11873 raft_consensus.cc:702] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:38.607153 11873 consensus_queue.cc:231] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.607235 11873 ts_tablet_manager.cc:1132] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.607281 11873 tablet_bootstrap.cc:439] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:38.607322 11768 tablet_bootstrap.cc:590] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:38.607388 11765 tablet_bootstrap.cc:590] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:38.607564 11873 tablet_bootstrap.cc:590] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:38.608008 11765 tablet_bootstrap.cc:439] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:38.608055 11765 ts_tablet_manager.cc:1104] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:38.608130 11767 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:38.608161 11767 ts_tablet_manager.cc:1104] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I1121 10:21:38.608181 11765 raft_consensus.cc:337] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.608196 11765 raft_consensus.cc:363] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.608203 11765 raft_consensus.cc:702] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:38.608295 11767 raft_consensus.cc:337] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.608310 11767 raft_consensus.cc:363] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.608317 11767 raft_consensus.cc:702] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:38.608263 11765 consensus_queue.cc:231] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.608374 11767 consensus_queue.cc:231] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.608423 11765 ts_tablet_manager.cc:1132] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.608458 11767 ts_tablet_manager.cc:1132] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.608489 11767 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:38.608844 11767 tablet_bootstrap.cc:590] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:38.609277 11768 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:38.609313 11768 ts_tablet_manager.cc:1104] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:38.609442 11768 raft_consensus.cc:337] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.609468 11768 raft_consensus.cc:363] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.609477 11768 raft_consensus.cc:702] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:38.609540 11768 consensus_queue.cc:231] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.609619 11768 ts_tablet_manager.cc:1132] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.609658 11768 tablet_bootstrap.cc:439] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:38.609925 11873 tablet_bootstrap.cc:439] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:38.609953 11767 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:38.609957 11873 ts_tablet_manager.cc:1104] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:38.609978 11768 tablet_bootstrap.cc:590] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:38.609988 11767 ts_tablet_manager.cc:1104] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:38.610117 11767 raft_consensus.cc:337] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.610123 11873 raft_consensus.cc:337] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.610146 11767 raft_consensus.cc:363] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.610154 11767 raft_consensus.cc:702] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:38.610157 11873 raft_consensus.cc:363] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.610167 11873 raft_consensus.cc:702] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:38.610240 11873 consensus_queue.cc:231] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.610350 11767 consensus_queue.cc:231] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.610373 11873 ts_tablet_manager.cc:1132] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:38.610492 11768 tablet_bootstrap.cc:439] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:38.610518 11768 ts_tablet_manager.cc:1104] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:21:38.610527 11767 ts_tablet_manager.cc:1132] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:38.610657 11768 raft_consensus.cc:337] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.610671 11768 raft_consensus.cc:363] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.610677 11768 raft_consensus.cc:702] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:38.610734 11768 consensus_queue.cc:231] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.610790 11768 ts_tablet_manager.cc:1132] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.638515 12100 raft_consensus.cc:469] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:38.638648 12100 raft_consensus.cc:491] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.638744 12100 leader_election.cc:296] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:38.638952 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "40e235c631e14bf8b4fe83e2953d8c87" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:38.638991 10953 raft_consensus.cc:2359] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 0. I1121 10:21:38.638998 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "40e235c631e14bf8b4fe83e2953d8c87" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:38.639032 11070 raft_consensus.cc:2359] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 0. I1121 10:21:38.639133 10764 leader_election.cc:310] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:38.639324 12100 raft_consensus.cc:2653] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:38.639345 12100 raft_consensus.cc:469] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:38.639353 12100 raft_consensus.cc:2905] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.640027 12100 raft_consensus.cc:491] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.640161 12100 leader_election.cc:296] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:38.640297 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "40e235c631e14bf8b4fe83e2953d8c87" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:38.640329 11070 raft_consensus.cc:2905] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.640653 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "40e235c631e14bf8b4fe83e2953d8c87" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:38.640731 10953 raft_consensus.cc:2905] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.641144 11070 raft_consensus.cc:2359] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 1. I1121 10:21:38.641307 10762 leader_election.cc:310] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:38.641435 12100 raft_consensus.cc:2653] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:38.641453 10953 raft_consensus.cc:2359] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 1. I1121 10:21:38.641484 12100 raft_consensus.cc:664] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Becoming Leader. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Running, Role: LEADER I1121 10:21:38.641551 12100 consensus_queue.cc:208] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } I1121 10:21:38.642114 10708 catalog_manager.cc:4143] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: term changed from 0 to 1, leader changed from to cd693d36f0014dc3a7c3f26196124794 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } } I1121 10:21:38.669708 12058 raft_consensus.cc:469] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:38.669828 12058 raft_consensus.cc:491] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.670114 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "114dbe156d634fca957db3164f8afabb" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:38.670182 10815 raft_consensus.cc:2359] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:38.670223 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "114dbe156d634fca957db3164f8afabb" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:38.670258 11204 raft_consensus.cc:2359] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 0. I1121 10:21:38.670408 10890 leader_election.cc:310] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:38.670488 12067 raft_consensus.cc:2653] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:38.670503 12067 raft_consensus.cc:469] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:38.670511 12067 raft_consensus.cc:2905] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.670603 12058 leader_election.cc:296] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:38.671273 12067 raft_consensus.cc:491] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.671339 12067 leader_election.cc:296] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Requested vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:38.671488 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "114dbe156d634fca957db3164f8afabb" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:38.671519 10815 raft_consensus.cc:2905] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.671521 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "114dbe156d634fca957db3164f8afabb" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:38.671545 11204 raft_consensus.cc:2905] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.672220 11204 raft_consensus.cc:2359] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:38.672248 10815 raft_consensus.cc:2359] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 1. I1121 10:21:38.672399 10890 leader_election.cc:310] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044; no voters: I1121 10:21:38.672478 12067 raft_consensus.cc:2653] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:38.672560 12067 raft_consensus.cc:664] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:38.672627 12067 consensus_queue.cc:208] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } I1121 10:21:38.673247 10708 catalog_manager.cc:4143] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 0 to 1, leader changed from to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } } I1121 10:21:38.761574 11986 raft_consensus.cc:469] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:38.761677 11986 raft_consensus.cc:491] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.761862 11986 leader_election.cc:296] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:38.762106 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ef99b04fd634cd599c435ad3383c4d5" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:38.762106 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ef99b04fd634cd599c435ad3383c4d5" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:38.762156 10953 raft_consensus.cc:2359] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:38.762156 11070 raft_consensus.cc:2359] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 0. I1121 10:21:38.762358 11135 leader_election.cc:310] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:38.762428 11986 raft_consensus.cc:2653] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:38.762452 11986 raft_consensus.cc:469] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:38.762461 11986 raft_consensus.cc:2905] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.763144 11986 raft_consensus.cc:491] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.763223 11986 leader_election.cc:296] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:38.763387 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ef99b04fd634cd599c435ad3383c4d5" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:38.763417 10953 raft_consensus.cc:2905] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.763428 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ef99b04fd634cd599c435ad3383c4d5" candidate_uuid: "5457a73f9ecb40ee9889513a8412937b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:38.763470 11070 raft_consensus.cc:2905] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.764063 11070 raft_consensus.cc:2359] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:38.764070 10953 raft_consensus.cc:2359] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5457a73f9ecb40ee9889513a8412937b in term 1. I1121 10:21:38.764247 11135 leader_election.cc:310] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:38.764335 11986 raft_consensus.cc:2653] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:38.764428 11986 raft_consensus.cc:664] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Becoming Leader. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Running, Role: LEADER I1121 10:21:38.764487 11986 consensus_queue.cc:208] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.765089 10708 catalog_manager.cc:4143] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: term changed from 0 to 1, leader changed from to 5457a73f9ecb40ee9889513a8412937b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } } W1121 10:21:38.845036 12120 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_8' I1121 10:21:38.873273 10815 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:38.873517 11204 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:38.873693 11070 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:38.873735 10953 raft_consensus.cc:1181] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:38.873775 12067 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.873832 12058 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.873898 10953 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:38.873989 12101 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.874092 12079 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.874105 11070 raft_consensus.cc:1181] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:38.874125 12118 consensus_queue.cc:984] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:38.874464 11986 consensus_queue.cc:984] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 23.0397 ms time per row: 0.0115198 ms I1121 10:21:38.888237 10707 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60768: name: "rebalance_test_table_9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:21:38.889699 11185 tablet_service.cc:1023] Processing CreateTablet for tablet 69ac58ad89e447a494fc407037d27f8d (table=rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.889762 10935 tablet_service.cc:1023] Processing CreateTablet for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 (table=rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.889842 11185 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69ac58ad89e447a494fc407037d27f8d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.889866 10935 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.889912 11061 tablet_service.cc:1023] Processing CreateTablet for tablet 69ac58ad89e447a494fc407037d27f8d (table=rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.890012 11061 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69ac58ad89e447a494fc407037d27f8d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.890852 11768 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044: Bootstrap starting. I1121 10:21:38.891119 11180 tablet_service.cc:1023] Processing CreateTablet for tablet cc935fc8b0984388a5db91f6e37824fa (table=rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.891172 11768 tablet_bootstrap.cc:590] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044: No blocks or log segments found. Creating new log. I1121 10:21:38.891201 11180 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet cc935fc8b0984388a5db91f6e37824fa. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.891194 11873 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:38.891211 11060 tablet_service.cc:1023] Processing CreateTablet for tablet cc935fc8b0984388a5db91f6e37824fa (table=rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.891278 11060 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet cc935fc8b0984388a5db91f6e37824fa. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.891512 11873 tablet_bootstrap.cc:590] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:38.891993 11873 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:38.892031 11873 ts_tablet_manager.cc:1104] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:38.892168 11873 raft_consensus.cc:337] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.892190 11873 raft_consensus.cc:363] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.892199 11873 raft_consensus.cc:702] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:38.892235 11765 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:38.892249 11873 consensus_queue.cc:231] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.892321 11059 tablet_service.cc:1023] Processing CreateTablet for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 (table=rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.892419 11059 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.892613 10805 tablet_service.cc:1023] Processing CreateTablet for tablet 69ac58ad89e447a494fc407037d27f8d (table=rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.892736 10805 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69ac58ad89e447a494fc407037d27f8d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.892776 11765 tablet_bootstrap.cc:590] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:38.893667 11767 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:38.893698 10807 tablet_service.cc:1023] Processing CreateTablet for tablet cc935fc8b0984388a5db91f6e37824fa (table=rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.893762 10807 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet cc935fc8b0984388a5db91f6e37824fa. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.893996 11767 tablet_bootstrap.cc:590] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:38.894505 11767 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:38.894544 11767 ts_tablet_manager.cc:1104] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:38.894681 11767 raft_consensus.cc:337] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.894707 11767 raft_consensus.cc:363] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.894728 11767 raft_consensus.cc:702] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:38.894795 11767 consensus_queue.cc:231] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.894805 10804 tablet_service.cc:1023] Processing CreateTablet for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 (table=rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:21:38.894867 10804 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:38.892333 11873 ts_tablet_manager.cc:1132] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.896358 11873 tablet_bootstrap.cc:439] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:38.894873 11767 ts_tablet_manager.cc:1132] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.896597 11767 tablet_bootstrap.cc:439] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:38.896847 11767 tablet_bootstrap.cc:590] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:38.897115 11873 tablet_bootstrap.cc:590] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:38.897588 11768 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044: No bootstrap required, opened a new log I1121 10:21:38.897629 11768 ts_tablet_manager.cc:1104] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s I1121 10:21:38.897675 11765 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:38.897722 11765 ts_tablet_manager.cc:1104] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s I1121 10:21:38.897786 11768 raft_consensus.cc:337] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.897806 11768 raft_consensus.cc:363] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.897814 11768 raft_consensus.cc:702] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Initialized, Role: FOLLOWER I1121 10:21:38.897871 11768 consensus_queue.cc:231] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.897872 11765 raft_consensus.cc:337] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.897889 11765 raft_consensus.cc:363] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.897897 11765 raft_consensus.cc:702] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:38.897940 11767 tablet_bootstrap.cc:439] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:38.897960 11765 consensus_queue.cc:231] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:38.897964 11768 ts_tablet_manager.cc:1132] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.897975 11767 ts_tablet_manager.cc:1104] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:21:38.898039 11765 ts_tablet_manager.cc:1132] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.898087 11765 tablet_bootstrap.cc:439] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b: Bootstrap starting. I1121 10:21:38.898102 11767 raft_consensus.cc:337] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.898128 11767 raft_consensus.cc:363] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.898135 11767 raft_consensus.cc:702] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:38.898207 11767 consensus_queue.cc:231] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.898285 11767 ts_tablet_manager.cc:1132] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.898327 11767 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794: Bootstrap starting. I1121 10:21:38.898414 11765 tablet_bootstrap.cc:590] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b: No blocks or log segments found. Creating new log. I1121 10:21:38.898646 11767 tablet_bootstrap.cc:590] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794: No blocks or log segments found. Creating new log. I1121 10:21:38.899003 11765 tablet_bootstrap.cc:439] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b: No bootstrap required, opened a new log I1121 10:21:38.899041 11765 ts_tablet_manager.cc:1104] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:38.899180 11873 tablet_bootstrap.cc:439] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:38.899191 11765 raft_consensus.cc:337] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.899204 11873 ts_tablet_manager.cc:1104] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:38.899206 11765 raft_consensus.cc:363] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.899214 11765 raft_consensus.cc:702] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Initialized, Role: FOLLOWER I1121 10:21:38.899271 11765 consensus_queue.cc:231] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.899333 11765 ts_tablet_manager.cc:1132] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.899361 11873 raft_consensus.cc:337] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.899376 11873 raft_consensus.cc:363] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.899384 11873 raft_consensus.cc:702] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:38.899443 11873 consensus_queue.cc:231] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.899515 11873 ts_tablet_manager.cc:1132] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.899560 11873 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: Bootstrap starting. I1121 10:21:38.899833 11873 tablet_bootstrap.cc:590] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: No blocks or log segments found. Creating new log. I1121 10:21:38.900463 11767 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794: No bootstrap required, opened a new log I1121 10:21:38.900507 11767 ts_tablet_manager.cc:1104] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s I1121 10:21:38.900792 11767 raft_consensus.cc:337] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.900817 11767 raft_consensus.cc:363] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.900823 11767 raft_consensus.cc:702] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Initialized, Role: FOLLOWER I1121 10:21:38.900876 11767 consensus_queue.cc:231] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.900949 11767 ts_tablet_manager.cc:1132] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.901136 11873 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: No bootstrap required, opened a new log I1121 10:21:38.901171 11873 ts_tablet_manager.cc:1104] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:21:38.901324 11873 raft_consensus.cc:337] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.901347 11873 raft_consensus.cc:363] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:21:38.901356 11873 raft_consensus.cc:702] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Initialized, Role: FOLLOWER I1121 10:21:38.901417 11873 consensus_queue.cc:231] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.901537 11873 ts_tablet_manager.cc:1132] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:38.993927 12100 raft_consensus.cc:469] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:38.994015 12100 raft_consensus.cc:491] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.994138 12100 leader_election.cc:296] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:38.994396 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "cc935fc8b0984388a5db91f6e37824fa" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:38.994439 11204 raft_consensus.cc:2359] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 0. I1121 10:21:38.994484 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "cc935fc8b0984388a5db91f6e37824fa" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" is_pre_election: true I1121 10:21:38.994526 11070 raft_consensus.cc:2359] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 0. I1121 10:21:38.994565 10765 leader_election.cc:310] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:38.994643 12100 raft_consensus.cc:2653] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:38.994665 12100 raft_consensus.cc:469] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:38.994673 12100 raft_consensus.cc:2905] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.995434 12100 raft_consensus.cc:491] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.995504 12100 leader_election.cc:296] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Requested vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:38.995718 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "cc935fc8b0984388a5db91f6e37824fa" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:38.995748 11204 raft_consensus.cc:2905] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.995757 11070 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "cc935fc8b0984388a5db91f6e37824fa" candidate_uuid: "cd693d36f0014dc3a7c3f26196124794" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:38.995781 11070 raft_consensus.cc:2905] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:38.996286 11070 raft_consensus.cc:2359] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 1. I1121 10:21:38.996299 11204 raft_consensus.cc:2359] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd693d36f0014dc3a7c3f26196124794 in term 1. I1121 10:21:38.996423 10762 leader_election.cc:310] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:38.996506 12100 raft_consensus.cc:2653] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:38.996587 12100 raft_consensus.cc:664] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Becoming Leader. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Running, Role: LEADER I1121 10:21:38.996661 12100 consensus_queue.cc:208] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:38.997287 10707 catalog_manager.cc:4143] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 reported cstate change: term changed from 0 to 1, leader changed from to cd693d36f0014dc3a7c3f26196124794 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } } I1121 10:21:39.053221 12102 raft_consensus.cc:469] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:39.053356 12102 raft_consensus.cc:491] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:39.053458 12102 leader_election.cc:296] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:39.053694 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "69ac58ad89e447a494fc407037d27f8d" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" is_pre_election: true I1121 10:21:39.053745 11204 raft_consensus.cc:2359] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:39.053740 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "69ac58ad89e447a494fc407037d27f8d" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:39.053774 10815 raft_consensus.cc:2359] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:39.053978 11014 leader_election.cc:310] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:39.054091 12102 raft_consensus.cc:2653] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:39.054116 12102 raft_consensus.cc:469] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:39.054124 12102 raft_consensus.cc:2905] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:39.054872 12102 raft_consensus.cc:491] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:39.055024 12102 leader_election.cc:296] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:39.055138 11204 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "69ac58ad89e447a494fc407037d27f8d" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:39.055171 11204 raft_consensus.cc:2905] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:39.055166 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "69ac58ad89e447a494fc407037d27f8d" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:39.055188 10815 raft_consensus.cc:2905] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:39.055717 10815 raft_consensus.cc:2359] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:39.055730 11204 raft_consensus.cc:2359] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:39.055887 11011 leader_election.cc:310] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:39.055956 12102 raft_consensus.cc:2653] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:39.055990 12102 raft_consensus.cc:664] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:39.056059 12102 consensus_queue.cc:208] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } I1121 10:21:39.056722 10707 catalog_manager.cc:4143] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } } I1121 10:21:39.062386 12102 raft_consensus.cc:469] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:21:39.062451 12102 raft_consensus.cc:491] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:39.062572 12102 leader_election.cc:296] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:39.062723 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" is_pre_election: true I1121 10:21:39.062816 10954 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" is_pre_election: true I1121 10:21:39.062845 10815 raft_consensus.cc:2359] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:39.062851 10954 raft_consensus.cc:2359] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 0. I1121 10:21:39.062981 11013 leader_election.cc:310] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:39.063063 12102 raft_consensus.cc:2653] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:21:39.063089 12102 raft_consensus.cc:469] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:21:39.063098 12102 raft_consensus.cc:2905] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:39.063611 12102 raft_consensus.cc:491] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:39.063769 12102 leader_election.cc:296] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:39.063844 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:39.063851 10954 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:39.063861 10815 raft_consensus.cc:2905] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:39.063876 10954 raft_consensus.cc:2905] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:21:39.064450 10954 raft_consensus.cc:2359] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:39.064659 10815 raft_consensus.cc:2359] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 1. I1121 10:21:39.064752 11013 leader_election.cc:310] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:39.064896 12102 raft_consensus.cc:2653] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:21:39.064934 12102 raft_consensus.cc:664] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:39.065032 12102 consensus_queue.cc:208] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } I1121 10:21:39.065601 10707 catalog_manager.cc:4143] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 0 to 1, leader changed from to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } } W1121 10:21:39.128859 12157 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_9' I1121 10:21:39.154929 11070 raft_consensus.cc:1181] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:39.154937 11204 raft_consensus.cc:1181] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:39.155166 12079 consensus_queue.cc:984] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:39.155341 12079 consensus_queue.cc:984] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:39.155443 10815 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:39.155515 10816 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:39.155663 12175 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:39.155954 11204 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:21:39.156268 10954 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:21:39.156567 12172 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:39.158294 12172 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:39.158782 12172 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:21:39.163120 11818 mvcc.cc:203] Tried to move safe_time back from 6448462639739146240 to 6448462639333883904. Current Snapshot: MvccSnapshot[committed={T|T < 6448462639739146240 or (T in {6448462639739146240})}] Generator report time total : 21.7895 ms time per row: 0.0108947 ms I1121 10:21:39.169389 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.70.5:35757 --local_ip_for_outbound_sockets=127.5.70.5 --webserver_port=40163 --webserver_interface=127.5.70.5 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:39.207291 12187 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:39.207445 12187 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:39.208367 12187 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:39.208389 12187 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:39.208403 12187 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.5 I1121 10:21:39.209625 12187 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.5:35757 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.70.5 --webserver_port=40163 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.12187 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.5 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:39.211359 12187 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:39.212502 12187 hybrid_clock.cc:183] HybridClock initialized: now 1574331699212475 us; error 32 us; skew 500 ppm I1121 10:21:39.212695 12187 fs_manager.cc:261] Metadata directory not provided I1121 10:21:39.212754 12187 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:39.213932 12187 fs_manager.cc:404] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:21:39.214112 12187 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:39.214241 12187 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:39.215178 12187 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:39.215622 12187 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal uuid: "fed6a935d10042b48cdf1f30824f6c57" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:39.215682 12187 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:39.252123 12187 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:39.252507 12187 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:39.252560 12187 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:39.258405 12187 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.5:35757 I1121 10:21:39.258733 12187 webserver.cc:382] Webserver started at http://127.5.70.5:40163/ using document root and password file I1121 10:21:39.259018 12187 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb I1121 10:21:39.263228 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 12187 I1121 10:21:39.263438 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.70.6:37787 --local_ip_for_outbound_sockets=127.5.70.6 --webserver_port=42409 --webserver_interface=127.5.70.6 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} I1121 10:21:39.265597 12308 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:39.265663 12308 heartbeater.cc:406] Registering TS with master... I1121 10:21:39.265792 12308 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:39.266167 10707 ts_manager.cc:192] Re-registered known tserver with Master: fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757) I1121 10:21:39.267047 10707 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.5:33859 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:39.305336 12311 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:39.305491 12311 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:39.306255 12311 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:39.306274 12311 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:39.306293 12311 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.6 I1121 10:21:39.307468 12311 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.6:37787 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.70.6 --webserver_port=42409 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.12311 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.6 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:39.309337 12311 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:39.310542 12311 hybrid_clock.cc:183] HybridClock initialized: now 1574331699310519 us; error 35 us; skew 500 ppm I1121 10:21:39.310658 12311 fs_manager.cc:261] Metadata directory not provided I1121 10:21:39.310699 12311 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:39.311993 12311 fs_manager.cc:404] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:21:39.312139 12311 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:39.312288 12311 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:39.312974 12311 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:39.313390 12311 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal uuid: "2c3332f15c014e68bed03f69d8c6287d" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:39.313437 12311 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:39.329111 12311 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:39.329442 12311 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:39.329460 12311 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:39.334882 12311 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.6:37787 I1121 10:21:39.335196 12311 webserver.cc:382] Webserver started at http://127.5.70.6:42409/ using document root and password file I1121 10:21:39.335453 12311 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb I1121 10:21:39.338385 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 12311 I1121 10:21:39.338644 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.70.7:40193 --local_ip_for_outbound_sockets=127.5.70.7 --webserver_port=35805 --webserver_interface=127.5.70.7 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} I1121 10:21:39.340790 12432 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:39.340845 12432 heartbeater.cc:406] Registering TS with master... I1121 10:21:39.340975 12432 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:39.341332 10707 ts_manager.cc:192] Re-registered known tserver with Master: 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787) I1121 10:21:39.342155 10707 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.6:56027 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:39.379937 12435 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:39.380110 12435 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:39.380913 12435 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:39.380934 12435 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:39.380949 12435 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.7 I1121 10:21:39.382095 12435 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.7:40193 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.70.7 --webserver_port=35805 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.12435 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.7 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:39.383827 12435 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:39.384963 12435 hybrid_clock.cc:183] HybridClock initialized: now 1574331699384938 us; error 42 us; skew 500 ppm I1121 10:21:39.385093 12435 fs_manager.cc:261] Metadata directory not provided I1121 10:21:39.385136 12435 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:39.386152 12435 fs_manager.cc:404] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:21:39.386302 12435 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:39.386395 12435 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:39.387094 12435 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:39.387490 12435 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal uuid: "77830a7609a549e0a1c76586d0e301ce" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:39.387538 12435 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:39.424607 12435 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:39.425724 12435 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:39.425745 12435 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:39.432458 12435 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.7:40193 I1121 10:21:39.432824 12435 webserver.cc:382] Webserver started at http://127.5.70.7:35805/ using document root and password file I1121 10:21:39.433097 12435 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb I1121 10:21:39.433720 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 12435 I1121 10:21:39.433976 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data --block_manager=log --rpc_bind_addresses=127.5.70.8:41471 --local_ip_for_outbound_sockets=127.5.70.8 --webserver_port=41913 --webserver_interface=127.5.70.8 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} I1121 10:21:39.438252 12556 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:39.438314 12556 heartbeater.cc:406] Registering TS with master... I1121 10:21:39.438434 12556 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:39.438818 10707 ts_manager.cc:192] Re-registered known tserver with Master: 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193) I1121 10:21:39.439412 10707 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.7:47315 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:39.515161 12559 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:39.515329 12559 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:39.516234 12559 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:39.516258 12559 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:39.516278 12559 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.8 I1121 10:21:39.521833 12559 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.8:41471 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb --webserver_interface=127.5.70.8 --webserver_port=41913 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.12559 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.8 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:39.523763 12559 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:39.524914 12559 hybrid_clock.cc:183] HybridClock initialized: now 1574331699524889 us; error 29 us; skew 500 ppm I1121 10:21:39.525076 12559 fs_manager.cc:261] Metadata directory not provided I1121 10:21:39.525132 12559 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:39.526540 12559 fs_manager.cc:404] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:21:39.526796 12559 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:39.526937 12559 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:39.527972 12559 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:39.528662 12559 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" format_stamp: "Formatted at 2019-11-21 10:21:31 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:39.528749 12559 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:39.572168 12559 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:39.572647 12559 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:39.572667 12559 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:39.578578 12559 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.8:41471 I1121 10:21:39.578917 12559 webserver.cc:382] Webserver started at http://127.5.70.8:41913/ using document root and password file I1121 10:21:39.579206 12559 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb I1121 10:21:39.583472 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 12559 I1121 10:21:39.583571 12680 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:39.583622 12680 heartbeater.cc:406] Registering TS with master... I1121 10:21:39.583739 12680 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:39.584064 10707 ts_manager.cc:192] Re-registered known tserver with Master: 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471) I1121 10:21:39.584808 10707 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.8:33569 W1121 10:21:39.698141 12684 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:39.874027 12806 flags.cc:404] Enabled unsafe flag: --never_fsync=true I1121 10:21:39.876138 10815 consensus_queue.cc:208] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:39.876896 10954 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.876952 11070 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.877146 12101 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.877255 12101 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.878239 12101 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:39.878484 11070 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:39.879704 10703 catalog_manager.cc:4143] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:39.880908 10954 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } W1121 10:21:39.884174 10763 consensus_peers.cc:473] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e0bc1c59a1b4e2c9e761a6de8085226. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.887823 11070 consensus_queue.cc:208] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:39.888371 10815 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.888388 10954 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.888628 12902 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.888703 12174 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.889442 12174 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:39.889816 10954 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:39.891089 10703 catalog_manager.cc:4143] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:39.891482 10815 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } W1121 10:21:39.892256 11012 consensus_peers.cc:473] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6765d5e95b8f45d4a8f0bb9e5d76b9f0. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.896370 11070 consensus_queue.cc:208] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:39.896924 11204 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.897167 12902 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.898066 10815 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.898260 12902 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.898753 12902 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:39.899210 10815 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } W1121 10:21:39.900115 11012 consensus_peers.cc:473] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a40d6cf45df24788adc85f238c9bcc7e. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.900430 10703 catalog_manager.cc:4143] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:39.902074 11204 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:39.905692 11070 consensus_queue.cc:208] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:39.906217 10954 raft_consensus.cc:1181] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.906356 10815 raft_consensus.cc:1181] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.906438 12184 consensus_queue.cc:984] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.906569 12184 consensus_queue.cc:984] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.907330 12184 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.907516 10815 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.908660 10703 catalog_manager.cc:4143] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.909247 10954 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } W1121 10:21:39.910065 11011 consensus_peers.cc:473] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51d8830759cc4842af19cc0d22bb8d65. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.915025 10954 consensus_queue.cc:208] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:39.915596 11070 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.915633 11204 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.915792 12067 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.915854 12949 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.917058 12950 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:39.917090 11070 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:39.917160 11204 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:39.918566 10710 catalog_manager.cc:4143] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } W1121 10:21:39.922112 10888 consensus_peers.cc:473] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69c25581de6e4945a167245a4dcbbfe0. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.930593 11070 consensus_queue.cc:208] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:39.931298 10815 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.931514 12174 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.932107 11204 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.932286 12184 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.934726 12184 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } W1121 10:21:39.935708 11012 consensus_peers.cc:473] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8963dd2bc2da4cacb9d5e5329680d7d7. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.936066 10708 catalog_manager.cc:4143] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:39.936484 11204 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:39.938314 10815 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:39.941109 10954 consensus_queue.cc:208] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:39.941593 11204 raft_consensus.cc:1181] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.941787 12067 consensus_queue.cc:984] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.942055 11070 raft_consensus.cc:1181] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.942234 12949 consensus_queue.cc:984] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.943560 12067 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:39.943699 11070 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } W1121 10:21:39.944666 10888 consensus_peers.cc:473] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45f0a219c25243fbade25321fcc1067a. This is attempt 1: this message will repeat every 5th retry. Runtime error: ksck discovered errors I1121 10:21:39.945011 11204 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:39.945035 10708 catalog_manager.cc:4143] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:39.950342 10815 consensus_queue.cc:208] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:39.950834 11204 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.950881 10954 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.951027 12891 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.951146 12884 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.951915 12101 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.952044 10954 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.953564 10710 catalog_manager.cc:4143] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } W1121 10:21:39.953963 10762 consensus_peers.cc:473] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7de6e746a8246f89d8b65b59803c3af. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.954324 11204 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.957654 10954 consensus_queue.cc:208] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:39.958184 11204 raft_consensus.cc:1181] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.958209 11070 raft_consensus.cc:1181] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:39.958338 10888 consensus_peers.cc:473] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ec67ec228044beda81b53a3e842af49. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.958397 12981 consensus_queue.cc:984] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.958477 12067 consensus_queue.cc:984] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.959530 12067 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:39.959748 11204 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:39.960794 10708 catalog_manager.cc:4143] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:39.960794 11070 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:39.968101 10954 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:39.968639 10815 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.968768 11070 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.968870 12981 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.968948 12981 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s W1121 10:21:39.971715 10888 consensus_peers.cc:473] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ecac46041da45aba6e82df02354fe89. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.973024 12950 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:39.973243 10815 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:39.973309 11070 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:39.974684 10710 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:39.979616 10954 consensus_queue.cc:208] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:39.980185 11204 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.980378 12981 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.980305 10815 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.980608 12981 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.981185 13007 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.981339 11204 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.982425 10704 catalog_manager.cc:4143] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W1121 10:21:39.984042 10887 consensus_peers.cc:473] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25b55ff241d8442897ea97523a2cd6fd. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.985182 10815 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.988413 11070 consensus_queue.cc:208] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:39.989066 10954 raft_consensus.cc:1181] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:39.989095 11012 consensus_peers.cc:473] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 204d260bb58b494091ae91cb57a88ca8. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.989279 10815 raft_consensus.cc:1181] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.989401 12923 consensus_queue.cc:984] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.989457 12902 consensus_queue.cc:984] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.990334 12923 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:39.990387 10954 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:39.991544 10708 catalog_manager.cc:4143] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:39.991982 10815 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:39.997980 10815 consensus_queue.cc:208] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:39.998492 10954 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.998545 11070 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:39.998719 12101 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s W1121 10:21:39.998780 10762 consensus_peers.cc:473] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40e235c631e14bf8b4fe83e2953d8c87. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:39.998847 12891 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:39.999493 12101 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.999686 10954 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:39.999845 11070 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:40.001075 10704 catalog_manager.cc:4143] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:40.005630 11204 consensus_queue.cc:208] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:40.006232 10954 raft_consensus.cc:1181] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:40.006443 13039 consensus_queue.cc:984] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.006459 10815 raft_consensus.cc:1181] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:40.006690 13038 consensus_queue.cc:984] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.007144 12118 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:40.008447 10709 catalog_manager.cc:4143] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:40.008589 10954 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } W1121 10:21:40.009747 11136 consensus_peers.cc:473] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc68d4b786cd485285707592d3c25f80. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:40.011184 10815 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:40.015352 10815 consensus_queue.cc:208] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:40.015786 11204 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:40.015900 11070 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:40.015929 10763 consensus_peers.cc:473] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 702635e315b940138670eeb5ae6ace1b. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:40.015988 12891 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.016058 12891 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.016927 12884 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:40.017109 11070 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:40.017114 11205 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:40.018478 10708 catalog_manager.cc:4143] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:40.022641 11070 consensus_queue.cc:208] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:40.023136 11205 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:40.023136 10954 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:40.023254 11012 consensus_peers.cc:473] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b34e50304ff46e59622de2e979ac7b8. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:40.023373 12923 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.023448 12923 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.025048 12174 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:40.025194 11205 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:40.025446 10954 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:40.026434 10709 catalog_manager.cc:4143] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:40.031734 10954 consensus_queue.cc:208] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:40.032284 10815 raft_consensus.cc:1181] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:40.032335 10888 consensus_peers.cc:473] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5becd9fee1eb40cf8eebb4ffac223e15. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:40.032502 12950 consensus_queue.cc:984] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.032583 11070 raft_consensus.cc:1181] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:40.032749 12949 consensus_queue.cc:984] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.037065 12067 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:40.037339 11070 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:40.037344 10815 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:40.038605 10708 catalog_manager.cc:4143] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W1121 10:21:40.038808 12998 flags.cc:404] Enabled unsafe flag: --never_fsync=true I1121 10:21:40.043972 11070 consensus_queue.cc:208] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:40.044427 10954 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:40.044592 10815 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:40.044610 11011 consensus_peers.cc:473] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa85115a8bbd45dbb7ce1f18b923022a. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:40.044762 12902 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.044822 12923 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.048990 12923 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:40.049134 10815 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:40.049248 10954 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:40.050420 10709 catalog_manager.cc:4143] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:40.054996 10954 consensus_queue.cc:208] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:40.055485 11205 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:40.055557 10815 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:40.055665 10888 consensus_peers.cc:473] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 114dbe156d634fca957db3164f8afabb. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:40.055863 12950 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.055941 13007 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.056830 12950 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:40.057140 10815 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:40.058109 10710 catalog_manager.cc:4143] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:40.058431 11205 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:40.062232 11205 consensus_queue.cc:208] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:40.062733 10954 raft_consensus.cc:1181] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:40.062732 11070 raft_consensus.cc:1181] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:40.062930 13038 consensus_queue.cc:984] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.063431 13038 consensus_queue.cc:984] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:40.064136 12118 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:40.065433 11080 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:40.065448 10709 catalog_manager.cc:4143] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W1121 10:21:40.065974 11136 consensus_peers.cc:473] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ef99b04fd634cd599c435ad3383c4d5. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:40.066779 10954 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } W1121 10:21:40.179946 13188 flags.cc:404] Enabled unsafe flag: --never_fsync=true I1121 10:21:40.267879 12308 heartbeater.cc:447] Master 127.5.70.62:45301 was elected leader, sending a full tablet report... W1121 10:21:40.300825 13275 flags.cc:404] Enabled unsafe flag: --never_fsync=true I1121 10:21:40.342931 12432 heartbeater.cc:447] Master 127.5.70.62:45301 was elected leader, sending a full tablet report... I1121 10:21:40.398573 13378 ts_tablet_manager.cc:721] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:40.398634 13378 tablet_copy_client.cc:243] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:43035 I1121 10:21:40.399906 13380 ts_tablet_manager.cc:721] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:40.399940 13380 tablet_copy_client.cc:243] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:40.402865 10845 tablet_copy_service.cc:135] P cd693d36f0014dc3a7c3f26196124794: Received BeginTabletCopySession request for tablet c7de6e746a8246f89d8b65b59803c3af from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:38887) I1121 10:21:40.402915 10845 tablet_copy_service.cc:156] P cd693d36f0014dc3a7c3f26196124794: Beginning new tablet copy session on tablet c7de6e746a8246f89d8b65b59803c3af from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:38887: session id = fed6a935d10042b48cdf1f30824f6c57-c7de6e746a8246f89d8b65b59803c3af I1121 10:21:40.403460 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 51d8830759cc4842af19cc0d22bb8d65 from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:36527) I1121 10:21:40.403492 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 51d8830759cc4842af19cc0d22bb8d65 from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:36527: session id = fed6a935d10042b48cdf1f30824f6c57-51d8830759cc4842af19cc0d22bb8d65 I1121 10:21:40.403672 10845 tablet_copy_source_session.cc:204] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.404109 11085 tablet_copy_source_session.cc:204] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.404779 13387 ts_tablet_manager.cc:721] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:40.404839 13387 tablet_copy_client.cc:243] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:40.405007 13378 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c7de6e746a8246f89d8b65b59803c3af. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.405108 13380 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 51d8830759cc4842af19cc0d22bb8d65. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.408427 13380 tablet_copy_client.cc:581] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.408582 13389 ts_tablet_manager.cc:721] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:40.408604 13378 tablet_copy_client.cc:581] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.408633 13389 tablet_copy_client.cc:243] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:40.408818 13378 tablet_copy_client.cc:544] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.409508 13378 tablet_copy_client.cc:755] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.409543 13380 tablet_copy_client.cc:544] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.410032 13380 tablet_copy_client.cc:755] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.411237 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 45f0a219c25243fbade25321fcc1067a from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:43411) I1121 10:21:40.411278 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 45f0a219c25243fbade25321fcc1067a from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:43411: session id = 77830a7609a549e0a1c76586d0e301ce-45f0a219c25243fbade25321fcc1067a I1121 10:21:40.411942 10977 tablet_copy_source_session.cc:204] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.412204 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet a40d6cf45df24788adc85f238c9bcc7e from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:54065) I1121 10:21:40.412250 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet a40d6cf45df24788adc85f238c9bcc7e from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:54065: session id = 77830a7609a549e0a1c76586d0e301ce-a40d6cf45df24788adc85f238c9bcc7e I1121 10:21:40.413051 13387 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 45f0a219c25243fbade25321fcc1067a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.413096 11085 tablet_copy_source_session.cc:204] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.413460 13389 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet a40d6cf45df24788adc85f238c9bcc7e. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.415736 13387 tablet_copy_client.cc:581] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.415916 13387 tablet_copy_client.cc:544] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.416069 13389 tablet_copy_client.cc:581] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.416149 13389 tablet_copy_client.cc:544] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.416556 13387 tablet_copy_client.cc:755] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.416793 13389 tablet_copy_client.cc:755] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.435822 13411 ts_tablet_manager.cc:721] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:40.435912 13411 tablet_copy_client.cc:243] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:40.439358 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 114dbe156d634fca957db3164f8afabb from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:37621) I1121 10:21:40.439389 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 114dbe156d634fca957db3164f8afabb from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:37621: session id = 2c3332f15c014e68bed03f69d8c6287d-114dbe156d634fca957db3164f8afabb I1121 10:21:40.439944 10977 tablet_copy_source_session.cc:204] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.440475 12556 heartbeater.cc:447] Master 127.5.70.62:45301 was elected leader, sending a full tablet report... I1121 10:21:40.440624 13418 ts_tablet_manager.cc:721] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:40.440681 13418 tablet_copy_client.cc:243] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:40.440829 13411 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 114dbe156d634fca957db3164f8afabb. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.440856 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 5becd9fee1eb40cf8eebb4ffac223e15 from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:43411) I1121 10:21:40.440874 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 5becd9fee1eb40cf8eebb4ffac223e15 from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:43411: session id = 77830a7609a549e0a1c76586d0e301ce-5becd9fee1eb40cf8eebb4ffac223e15 I1121 10:21:40.441290 10977 tablet_copy_source_session.cc:204] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.441653 13418 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5becd9fee1eb40cf8eebb4ffac223e15. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.442585 13418 tablet_copy_client.cc:581] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.442703 13418 tablet_copy_client.cc:544] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.443163 13418 tablet_copy_client.cc:755] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.443825 13411 tablet_copy_client.cc:581] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.443954 13411 tablet_copy_client.cc:544] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.444571 13411 tablet_copy_client.cc:755] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 W1121 10:21:40.445204 13347 flags.cc:404] Enabled unsafe flag: --never_fsync=true I1121 10:21:40.456683 13437 ts_tablet_manager.cc:721] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:40.456765 13437 tablet_copy_client.cc:243] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:37207 I1121 10:21:40.459605 11218 tablet_copy_service.cc:135] P 5457a73f9ecb40ee9889513a8412937b: Received BeginTabletCopySession request for tablet cc68d4b786cd485285707592d3c25f80 from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:39461) I1121 10:21:40.459635 11218 tablet_copy_service.cc:156] P 5457a73f9ecb40ee9889513a8412937b: Beginning new tablet copy session on tablet cc68d4b786cd485285707592d3c25f80 from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:39461: session id = 2c3332f15c014e68bed03f69d8c6287d-cc68d4b786cd485285707592d3c25f80 I1121 10:21:40.460115 11218 tablet_copy_source_session.cc:204] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.460568 13437 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet cc68d4b786cd485285707592d3c25f80. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.461699 13437 tablet_copy_client.cc:581] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.461802 13437 tablet_copy_client.cc:544] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.462455 13437 tablet_copy_client.cc:755] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.465603 13440 ts_tablet_manager.cc:721] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:40.465685 13440 tablet_copy_client.cc:243] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:43035 I1121 10:21:40.466496 13444 ts_tablet_manager.cc:721] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:40.466549 13444 tablet_copy_client.cc:243] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:40.468505 10845 tablet_copy_service.cc:135] P cd693d36f0014dc3a7c3f26196124794: Received BeginTabletCopySession request for tablet 3e0bc1c59a1b4e2c9e761a6de8085226 from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:48773) I1121 10:21:40.468564 10845 tablet_copy_service.cc:156] P cd693d36f0014dc3a7c3f26196124794: Beginning new tablet copy session on tablet 3e0bc1c59a1b4e2c9e761a6de8085226 from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:48773: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-3e0bc1c59a1b4e2c9e761a6de8085226 I1121 10:21:40.469085 10845 tablet_copy_source_session.cc:204] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.469316 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:51069) I1121 10:21:40.469344 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:51069: session id = 2c3332f15c014e68bed03f69d8c6287d-6765d5e95b8f45d4a8f0bb9e5d76b9f0 I1121 10:21:40.469825 11085 tablet_copy_source_session.cc:204] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.469880 13440 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 3e0bc1c59a1b4e2c9e761a6de8085226. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.470151 13444 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.471513 13444 tablet_copy_client.cc:581] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.471691 13444 tablet_copy_client.cc:544] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.472225 13444 tablet_copy_client.cc:755] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.473014 13452 ts_tablet_manager.cc:721] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:40.473064 13452 tablet_copy_client.cc:243] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:37207 I1121 10:21:40.473894 13454 ts_tablet_manager.cc:721] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:40.473939 13454 tablet_copy_client.cc:243] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:40.474155 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 204d260bb58b494091ae91cb57a88ca8 from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:54065) I1121 10:21:40.474197 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 204d260bb58b494091ae91cb57a88ca8 from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:54065: session id = 77830a7609a549e0a1c76586d0e301ce-204d260bb58b494091ae91cb57a88ca8 I1121 10:21:40.474710 11085 tablet_copy_source_session.cc:204] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.475019 13454 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 204d260bb58b494091ae91cb57a88ca8. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.475272 13455 ts_tablet_manager.cc:721] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:40.475365 13455 tablet_copy_client.cc:243] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:40.476286 13440 tablet_copy_client.cc:581] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.476408 13440 tablet_copy_client.cc:544] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.476779 13454 tablet_copy_client.cc:581] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.476893 13454 tablet_copy_client.cc:544] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.477329 13454 tablet_copy_client.cc:755] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.477640 13440 tablet_copy_client.cc:755] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.478724 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 25b55ff241d8442897ea97523a2cd6fd from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:57625) I1121 10:21:40.478760 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 25b55ff241d8442897ea97523a2cd6fd from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:57625: session id = fed6a935d10042b48cdf1f30824f6c57-25b55ff241d8442897ea97523a2cd6fd I1121 10:21:40.479297 10977 tablet_copy_source_session.cc:204] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.479756 13455 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 25b55ff241d8442897ea97523a2cd6fd. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.481401 11218 tablet_copy_service.cc:135] P 5457a73f9ecb40ee9889513a8412937b: Received BeginTabletCopySession request for tablet 4ef99b04fd634cd599c435ad3383c4d5 from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:35393) I1121 10:21:40.481428 11218 tablet_copy_service.cc:156] P 5457a73f9ecb40ee9889513a8412937b: Beginning new tablet copy session on tablet 4ef99b04fd634cd599c435ad3383c4d5 from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:35393: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-4ef99b04fd634cd599c435ad3383c4d5 I1121 10:21:40.483115 13455 tablet_copy_client.cc:581] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.483219 13455 tablet_copy_client.cc:544] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.483657 11218 tablet_copy_source_session.cc:204] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.483844 13455 tablet_copy_client.cc:755] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.484011 13452 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ef99b04fd634cd599c435ad3383c4d5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.486768 13452 tablet_copy_client.cc:581] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.487008 13452 tablet_copy_client.cc:544] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.487529 13452 tablet_copy_client.cc:755] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.487882 13466 ts_tablet_manager.cc:721] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:40.487921 13466 tablet_copy_client.cc:243] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:40.491122 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 4ecac46041da45aba6e82df02354fe89 from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:59669) I1121 10:21:40.491153 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 4ecac46041da45aba6e82df02354fe89 from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:59669: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-4ecac46041da45aba6e82df02354fe89 I1121 10:21:40.491705 10977 tablet_copy_source_session.cc:204] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.492014 13466 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ecac46041da45aba6e82df02354fe89. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.493438 13466 tablet_copy_client.cc:581] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.493530 13466 tablet_copy_client.cc:544] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.494029 13466 tablet_copy_client.cc:755] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.501459 13478 ts_tablet_manager.cc:721] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:40.501513 13478 tablet_copy_client.cc:243] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:40.501708 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 0ec67ec228044beda81b53a3e842af49 from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:37621) I1121 10:21:40.501751 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 0ec67ec228044beda81b53a3e842af49 from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:37621: session id = 2c3332f15c014e68bed03f69d8c6287d-0ec67ec228044beda81b53a3e842af49 I1121 10:21:40.502351 10977 tablet_copy_source_session.cc:204] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.504323 13479 ts_tablet_manager.cc:721] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:40.504364 13479 tablet_copy_client.cc:243] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:40.505086 13478 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0ec67ec228044beda81b53a3e842af49. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.506179 13478 tablet_copy_client.cc:581] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.506281 13478 tablet_copy_client.cc:544] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.506780 13478 tablet_copy_client.cc:755] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.507493 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 8963dd2bc2da4cacb9d5e5329680d7d7 from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:52727) I1121 10:21:40.507524 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 8963dd2bc2da4cacb9d5e5329680d7d7 from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:52727: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-8963dd2bc2da4cacb9d5e5329680d7d7 I1121 10:21:40.508080 11085 tablet_copy_source_session.cc:204] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.508430 13479 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8963dd2bc2da4cacb9d5e5329680d7d7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.509672 13479 tablet_copy_client.cc:581] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.509768 13479 tablet_copy_client.cc:544] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.510296 13479 tablet_copy_client.cc:755] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.511036 13480 ts_tablet_manager.cc:721] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:40.511091 13480 tablet_copy_client.cc:243] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:43035 Runtime error: ksck discovered errors I1121 10:21:40.511325 10845 tablet_copy_service.cc:135] P cd693d36f0014dc3a7c3f26196124794: Received BeginTabletCopySession request for tablet 40e235c631e14bf8b4fe83e2953d8c87 from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:38887) I1121 10:21:40.511358 10845 tablet_copy_service.cc:156] P cd693d36f0014dc3a7c3f26196124794: Beginning new tablet copy session on tablet 40e235c631e14bf8b4fe83e2953d8c87 from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:38887: session id = fed6a935d10042b48cdf1f30824f6c57-40e235c631e14bf8b4fe83e2953d8c87 I1121 10:21:40.511850 10845 tablet_copy_source_session.cc:204] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.512225 13480 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 40e235c631e14bf8b4fe83e2953d8c87. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.513475 13480 tablet_copy_client.cc:581] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.513564 13480 tablet_copy_client.cc:544] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.514091 13480 tablet_copy_client.cc:755] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.515167 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/data --block_manager=log --rpc_bind_addresses=127.5.70.9:0 --local_ip_for_outbound_sockets=127.5.70.9 --webserver_interface=127.5.70.9 --webserver_port=0 --tserver_master_addrs=127.5.70.62:45301 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34111 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true --tablet_copy_download_file_inject_latency_ms=1500 --follower_unavailable_considered_failed_sec=30 with env {} I1121 10:21:40.537878 13482 ts_tablet_manager.cc:721] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:40.537943 13482 tablet_copy_client.cc:243] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:40.538180 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 69c25581de6e4945a167245a4dcbbfe0 from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:37621) I1121 10:21:40.538213 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 69c25581de6e4945a167245a4dcbbfe0 from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:37621: session id = 2c3332f15c014e68bed03f69d8c6287d-69c25581de6e4945a167245a4dcbbfe0 I1121 10:21:40.538714 10977 tablet_copy_source_session.cc:204] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.539085 13482 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69c25581de6e4945a167245a4dcbbfe0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.540213 13482 tablet_copy_client.cc:581] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.540314 13482 tablet_copy_client.cc:544] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.541024 13482 tablet_copy_client.cc:755] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:21:40.556633 13481 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:21:40.556800 13481 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:21:40.557675 13481 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:21:40.557698 13481 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:21:40.557719 13481 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.9 I1121 10:21:40.559039 13481 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34111 --ntp_initial_sync_wait_secs=10 --time_source=builtin --follower_unavailable_considered_failed_sec=30 --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.9:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/data/info.pb --webserver_interface=127.5.70.9 --webserver_port=0 --tablet_copy_download_file_inject_latency_ms=1500 --tserver_master_addrs=127.5.70.62:45301 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.13481 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.9 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:21:40.560932 13481 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:21:40.562093 13481 hybrid_clock.cc:183] HybridClock initialized: now 1574331700562067 us; error 34 us; skew 500 ppm I1121 10:21:40.562304 13481 fs_manager.cc:261] Metadata directory not provided I1121 10:21:40.562348 13481 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:21:40.562453 13481 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:21:40.562474 13481 server_base.cc:453] Attempting to create new FS layout instead I1121 10:21:40.563550 13481 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/data/instance: uuid: "80d2057cd4814fd3bfe268270d1abdd4" format_stamp: "Formatted at 2019-11-21 10:21:40 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:40.563812 13481 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/wal/instance: uuid: "80d2057cd4814fd3bfe268270d1abdd4" format_stamp: "Formatted at 2019-11-21 10:21:40 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:40.565425 13481 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s I1121 10:21:40.565708 13481 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:21:40.565873 13481 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:21:40.566717 13481 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:21:40.566752 13481 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/wal uuid: "80d2057cd4814fd3bfe268270d1abdd4" format_stamp: "Formatted at 2019-11-21 10:21:40 on dist-test-slave-dist-test-slave-5k2q" I1121 10:21:40.566795 13481 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:21:40.571867 13490 ts_tablet_manager.cc:721] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:40.571928 13490 tablet_copy_client.cc:243] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:43035 I1121 10:21:40.572152 10845 tablet_copy_service.cc:135] P cd693d36f0014dc3a7c3f26196124794: Received BeginTabletCopySession request for tablet 702635e315b940138670eeb5ae6ace1b from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:48773) I1121 10:21:40.572193 10845 tablet_copy_service.cc:156] P cd693d36f0014dc3a7c3f26196124794: Beginning new tablet copy session on tablet 702635e315b940138670eeb5ae6ace1b from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:48773: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-702635e315b940138670eeb5ae6ace1b I1121 10:21:40.572758 10845 tablet_copy_source_session.cc:204] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.573918 13490 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 702635e315b940138670eeb5ae6ace1b. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.574756 13490 tablet_copy_client.cc:581] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.574856 13490 tablet_copy_client.cc:544] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.575388 13490 tablet_copy_client.cc:755] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.585682 12680 heartbeater.cc:447] Master 127.5.70.62:45301 was elected leader, sending a full tablet report... I1121 10:21:40.600224 13481 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:21:40.600625 13481 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:21:40.600644 13481 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:21:40.607662 13481 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.9:39291 I1121 10:21:40.608036 13481 webserver.cc:382] Webserver started at http://127.5.70.9:40971/ using document root and password file I1121 10:21:40.608335 13481 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/data/info.pb I1121 10:21:40.608781 13606 ts_tablet_manager.cc:721] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:40.608842 13606 tablet_copy_client.cc:243] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:40.609098 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 7b34e50304ff46e59622de2e979ac7b8 from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:54065) I1121 10:21:40.609131 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 7b34e50304ff46e59622de2e979ac7b8 from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:54065: session id = 77830a7609a549e0a1c76586d0e301ce-7b34e50304ff46e59622de2e979ac7b8 I1121 10:21:40.609447 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 13481 I1121 10:21:40.609782 11085 tablet_copy_source_session.cc:204] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.610146 13606 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7b34e50304ff46e59622de2e979ac7b8. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.611073 13606 tablet_copy_client.cc:581] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.611171 13606 tablet_copy_client.cc:544] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.611680 13606 tablet_copy_client.cc:755] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:40.612681 13604 heartbeater.cc:322] Connected to a master server at 127.5.70.62:45301 I1121 10:21:40.612756 13604 heartbeater.cc:406] Registering TS with master... I1121 10:21:40.612916 13604 heartbeater.cc:455] Master 127.5.70.62:45301 requested a full tablet report, sending... I1121 10:21:40.613272 10701 ts_manager.cc:192] Registered new tserver with Master: 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291) I1121 10:21:40.613814 10701 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.9:36681 I1121 10:21:40.669123 13628 ts_tablet_manager.cc:721] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:40.669185 13628 tablet_copy_client.cc:243] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:40.669387 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet fa85115a8bbd45dbb7ce1f18b923022a from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:36527) I1121 10:21:40.669427 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet fa85115a8bbd45dbb7ce1f18b923022a from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:36527: session id = fed6a935d10042b48cdf1f30824f6c57-fa85115a8bbd45dbb7ce1f18b923022a I1121 10:21:40.670065 11085 tablet_copy_source_session.cc:204] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:40.670424 13628 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet fa85115a8bbd45dbb7ce1f18b923022a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:40.671377 13628 tablet_copy_client.cc:581] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:40.671478 13628 tablet_copy_client.cc:544] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:40.672116 13628 tablet_copy_client.cc:755] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:41.614470 13604 heartbeater.cc:447] Master 127.5.70.62:45301 was elected leader, sending a full tablet report... I1121 10:21:41.909729 13378 tablet_copy_client.cc:420] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.910228 13380 tablet_copy_client.cc:420] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.910688 13378 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:41.910931 13380 tablet_bootstrap.cc:439] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:41.912045 13378 log.cc:810] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: Log is configured to *not* fsync() on all Append() calls I1121 10:21:41.915473 13380 tablet_bootstrap.cc:439] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.915582 13378 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.915702 13380 tablet_bootstrap.cc:439] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:41.915777 13378 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:41.915815 13380 ts_tablet_manager.cc:1104] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s I1121 10:21:41.915864 13378 ts_tablet_manager.cc:1104] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s I1121 10:21:41.916750 13387 tablet_copy_client.cc:420] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.916807 13378 raft_consensus.cc:337] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:41.916806 13380 raft_consensus.cc:337] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:41.916867 13378 raft_consensus.cc:702] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:41.916870 13380 raft_consensus.cc:702] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:41.916991 13380 consensus_queue.cc:231] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:41.916991 13378 consensus_queue.cc:231] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:41.917153 13389 tablet_copy_client.cc:420] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.917197 13380 ts_tablet_manager.cc:1132] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s I1121 10:21:41.917434 13378 ts_tablet_manager.cc:1132] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s I1121 10:21:41.917801 13387 tablet_bootstrap.cc:439] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:41.917836 13389 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:41.917934 10845 tablet_copy_service.cc:325] P cd693d36f0014dc3a7c3f26196124794: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-c7de6e746a8246f89d8b65b59803c3af received from {username='slave'} at 127.5.70.5:38887 I1121 10:21:41.917969 10845 tablet_copy_service.cc:413] P cd693d36f0014dc3a7c3f26196124794: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-c7de6e746a8246f89d8b65b59803c3af on tablet c7de6e746a8246f89d8b65b59803c3af with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:41.919541 13387 log.cc:810] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: Log is configured to *not* fsync() on all Append() calls I1121 10:21:41.919541 13389 log.cc:810] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: Log is configured to *not* fsync() on all Append() calls I1121 10:21:41.919747 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-51d8830759cc4842af19cc0d22bb8d65 received from {username='slave'} at 127.5.70.5:36527 I1121 10:21:41.919781 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-51d8830759cc4842af19cc0d22bb8d65 on tablet 51d8830759cc4842af19cc0d22bb8d65 with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:41.924260 13387 tablet_bootstrap.cc:439] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.924316 13389 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.924574 13387 tablet_bootstrap.cc:439] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:41.924602 13389 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:41.924762 13387 ts_tablet_manager.cc:1104] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s I1121 10:21:41.924762 13389 ts_tablet_manager.cc:1104] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s I1121 10:21:41.926144 13389 raft_consensus.cc:337] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:41.926149 13387 raft_consensus.cc:337] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:41.926224 13389 raft_consensus.cc:702] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:41.926235 13387 raft_consensus.cc:702] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:41.926322 13389 consensus_queue.cc:231] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:41.926322 13387 consensus_queue.cc:231] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:41.926559 13387 ts_tablet_manager.cc:1132] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I1121 10:21:41.926582 13389 ts_tablet_manager.cc:1132] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I1121 10:21:41.926846 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-45f0a219c25243fbade25321fcc1067a received from {username='slave'} at 127.5.70.7:43411 I1121 10:21:41.926874 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-45f0a219c25243fbade25321fcc1067a on tablet 45f0a219c25243fbade25321fcc1067a with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:41.926877 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-a40d6cf45df24788adc85f238c9bcc7e received from {username='slave'} at 127.5.70.7:54065 I1121 10:21:41.926896 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-a40d6cf45df24788adc85f238c9bcc7e on tablet a40d6cf45df24788adc85f238c9bcc7e with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:41.931541 12513 raft_consensus.cc:1123] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:41.943408 13418 tablet_copy_client.cc:420] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.944317 13418 tablet_bootstrap.cc:439] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:41.944805 13411 tablet_copy_client.cc:420] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.945528 13411 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:41.946852 13411 log.cc:810] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: Log is configured to *not* fsync() on all Append() calls I1121 10:21:41.948006 13418 tablet_bootstrap.cc:439] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.948209 13418 tablet_bootstrap.cc:439] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:41.948300 13418 ts_tablet_manager.cc:1104] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s I1121 10:21:41.948442 13418 raft_consensus.cc:337] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:41.948460 13418 raft_consensus.cc:702] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:41.948557 13418 consensus_queue.cc:231] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:41.948716 13418 ts_tablet_manager.cc:1132] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.000s user 0.002s sys 0.001s I1121 10:21:41.949025 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-5becd9fee1eb40cf8eebb4ffac223e15 received from {username='slave'} at 127.5.70.7:43411 I1121 10:21:41.949050 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-5becd9fee1eb40cf8eebb4ffac223e15 on tablet 5becd9fee1eb40cf8eebb4ffac223e15 with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:41.950377 13411 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.950614 13411 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:41.950717 13411 ts_tablet_manager.cc:1104] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s I1121 10:21:41.951493 13411 raft_consensus.cc:337] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:41.951542 13411 raft_consensus.cc:702] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:41.951624 13411 consensus_queue.cc:231] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:41.951794 13411 ts_tablet_manager.cc:1132] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s I1121 10:21:41.951995 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-114dbe156d634fca957db3164f8afabb received from {username='slave'} at 127.5.70.6:37621 I1121 10:21:41.952019 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-114dbe156d634fca957db3164f8afabb on tablet 114dbe156d634fca957db3164f8afabb with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:41.962604 13437 tablet_copy_client.cc:420] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.963296 13437 tablet_bootstrap.cc:439] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:41.966925 13437 tablet_bootstrap.cc:439] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.967118 13437 tablet_bootstrap.cc:439] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:41.967222 13437 ts_tablet_manager.cc:1104] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s I1121 10:21:41.967378 13437 raft_consensus.cc:337] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:41.967401 13437 raft_consensus.cc:702] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:41.967479 13437 consensus_queue.cc:231] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:41.967566 13437 ts_tablet_manager.cc:1132] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.000s user 0.000s sys 0.003s I1121 10:21:41.967860 11218 tablet_copy_service.cc:325] P 5457a73f9ecb40ee9889513a8412937b: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-cc68d4b786cd485285707592d3c25f80 received from {username='slave'} at 127.5.70.6:39461 I1121 10:21:41.967890 11218 tablet_copy_service.cc:413] P 5457a73f9ecb40ee9889513a8412937b: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-cc68d4b786cd485285707592d3c25f80 on tablet cc68d4b786cd485285707592d3c25f80 with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:41.972393 13444 tablet_copy_client.cc:420] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.973280 13444 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:41.976279 13444 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.976464 13444 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:41.976586 13444 ts_tablet_manager.cc:1104] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.004s I1121 10:21:41.976720 13444 raft_consensus.cc:337] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:41.976745 13444 raft_consensus.cc:702] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:41.976802 13444 consensus_queue.cc:231] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:41.976879 13444 ts_tablet_manager.cc:1132] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:41.977288 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-6765d5e95b8f45d4a8f0bb9e5d76b9f0 received from {username='slave'} at 127.5.70.6:51069 I1121 10:21:41.977313 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-6765d5e95b8f45d4a8f0bb9e5d76b9f0 on tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:41.977468 13454 tablet_copy_client.cc:420] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.977850 13440 tablet_copy_client.cc:420] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.978276 13454 tablet_bootstrap.cc:439] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:41.978528 13440 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:41.979763 13440 log.cc:810] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: Log is configured to *not* fsync() on all Append() calls I1121 10:21:41.981889 13454 tablet_bootstrap.cc:439] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.982074 13454 tablet_bootstrap.cc:439] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:41.982151 13454 ts_tablet_manager.cc:1104] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s I1121 10:21:41.982308 13454 raft_consensus.cc:337] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:41.982331 13454 raft_consensus.cc:702] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:41.982367 13454 consensus_queue.cc:231] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:41.982442 13454 ts_tablet_manager.cc:1132] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s I1121 10:21:41.982636 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-204d260bb58b494091ae91cb57a88ca8 received from {username='slave'} at 127.5.70.7:54065 I1121 10:21:41.982795 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-204d260bb58b494091ae91cb57a88ca8 on tablet 204d260bb58b494091ae91cb57a88ca8 with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:41.983723 13440 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.983888 13440 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:41.983989 13440 ts_tablet_manager.cc:1104] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s I1121 10:21:41.983999 13455 tablet_copy_client.cc:420] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.984777 12513 raft_consensus.cc:1123] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:41.984977 13455 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:41.985188 13440 raft_consensus.cc:337] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:41.985247 13440 raft_consensus.cc:702] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:41.985348 13440 consensus_queue.cc:231] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:41.985769 13440 ts_tablet_manager.cc:1132] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I1121 10:21:41.985973 10845 tablet_copy_service.cc:325] P cd693d36f0014dc3a7c3f26196124794: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-3e0bc1c59a1b4e2c9e761a6de8085226 received from {username='slave'} at 127.5.70.8:48773 I1121 10:21:41.986003 10845 tablet_copy_service.cc:413] P cd693d36f0014dc3a7c3f26196124794: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-3e0bc1c59a1b4e2c9e761a6de8085226 on tablet 3e0bc1c59a1b4e2c9e761a6de8085226 with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:41.987689 13452 tablet_copy_client.cc:420] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.988246 13452 tablet_bootstrap.cc:439] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:41.988770 13455 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.989035 13455 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:41.989167 13455 ts_tablet_manager.cc:1104] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s I1121 10:21:41.989318 13455 raft_consensus.cc:337] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:41.989341 13455 raft_consensus.cc:702] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:41.989396 13455 consensus_queue.cc:231] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:41.989467 13455 ts_tablet_manager.cc:1132] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:41.992417 13452 tablet_bootstrap.cc:439] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.992625 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-25b55ff241d8442897ea97523a2cd6fd received from {username='slave'} at 127.5.70.5:57625 I1121 10:21:41.992658 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-25b55ff241d8442897ea97523a2cd6fd on tablet 25b55ff241d8442897ea97523a2cd6fd with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:41.992758 13452 tablet_bootstrap.cc:439] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:41.992969 13452 ts_tablet_manager.cc:1104] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s I1121 10:21:41.993126 13452 raft_consensus.cc:337] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:41.993151 13452 raft_consensus.cc:702] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:41.993201 13452 consensus_queue.cc:231] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:41.993405 13452 ts_tablet_manager.cc:1132] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:41.993590 11218 tablet_copy_service.cc:325] P 5457a73f9ecb40ee9889513a8412937b: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-4ef99b04fd634cd599c435ad3383c4d5 received from {username='slave'} at 127.5.70.8:35393 I1121 10:21:41.993619 11218 tablet_copy_service.cc:413] P 5457a73f9ecb40ee9889513a8412937b: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-4ef99b04fd634cd599c435ad3383c4d5 on tablet 4ef99b04fd634cd599c435ad3383c4d5 with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:41.994138 13466 tablet_copy_client.cc:420] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:41.994669 13466 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:41.998240 13466 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:41.998396 13466 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:41.998474 13466 ts_tablet_manager.cc:1104] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I1121 10:21:41.998589 13466 raft_consensus.cc:337] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:41.998608 13466 raft_consensus.cc:702] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:41.998648 13466 consensus_queue.cc:231] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:41.998798 13466 ts_tablet_manager.cc:1132] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:41.999104 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-4ecac46041da45aba6e82df02354fe89 received from {username='slave'} at 127.5.70.8:59669 I1121 10:21:41.999125 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-4ecac46041da45aba6e82df02354fe89 on tablet 4ecac46041da45aba6e82df02354fe89 with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:42.005424 12389 raft_consensus.cc:1123] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.006909 13478 tablet_copy_client.cc:420] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:42.007661 13478 tablet_bootstrap.cc:439] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:42.010458 13479 tablet_copy_client.cc:420] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:42.011381 13479 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:42.011904 13478 tablet_bootstrap.cc:439] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:42.012162 13478 tablet_bootstrap.cc:439] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:42.012254 13478 ts_tablet_manager.cc:1104] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s I1121 10:21:42.012408 13478 raft_consensus.cc:337] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:42.012437 13478 raft_consensus.cc:702] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:42.012495 13478 consensus_queue.cc:231] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:42.012974 13478 ts_tablet_manager.cc:1132] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s I1121 10:21:42.013286 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-0ec67ec228044beda81b53a3e842af49 received from {username='slave'} at 127.5.70.6:37621 I1121 10:21:42.013314 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-0ec67ec228044beda81b53a3e842af49 on tablet 0ec67ec228044beda81b53a3e842af49 with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:42.014341 13480 tablet_copy_client.cc:420] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:42.015231 13480 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:42.016299 13479 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:42.016470 12513 raft_consensus.cc:1123] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.016693 13479 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:42.016796 13479 ts_tablet_manager.cc:1104] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s I1121 10:21:42.016974 13479 raft_consensus.cc:337] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:42.016996 13479 raft_consensus.cc:702] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:42.017060 13479 consensus_queue.cc:231] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:42.017374 13479 ts_tablet_manager.cc:1132] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:21:42.017571 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-8963dd2bc2da4cacb9d5e5329680d7d7 received from {username='slave'} at 127.5.70.8:52727 I1121 10:21:42.017601 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-8963dd2bc2da4cacb9d5e5329680d7d7 on tablet 8963dd2bc2da4cacb9d5e5329680d7d7 with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:42.020397 13480 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:42.020673 13480 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:42.020817 13480 ts_tablet_manager.cc:1104] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s I1121 10:21:42.020965 13480 raft_consensus.cc:337] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:42.020994 13480 raft_consensus.cc:702] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:42.021054 13480 consensus_queue.cc:231] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:42.021169 13480 ts_tablet_manager.cc:1132] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s I1121 10:21:42.021373 10845 tablet_copy_service.cc:325] P cd693d36f0014dc3a7c3f26196124794: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-40e235c631e14bf8b4fe83e2953d8c87 received from {username='slave'} at 127.5.70.5:38887 I1121 10:21:42.021399 10845 tablet_copy_service.cc:413] P cd693d36f0014dc3a7c3f26196124794: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-40e235c631e14bf8b4fe83e2953d8c87 on tablet 40e235c631e14bf8b4fe83e2953d8c87 with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:42.024281 12637 raft_consensus.cc:1123] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.041227 13482 tablet_copy_client.cc:420] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:42.042148 13482 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:42.046538 13482 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:42.046764 13482 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:42.046878 13482 ts_tablet_manager.cc:1104] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s I1121 10:21:42.047017 13482 raft_consensus.cc:337] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:42.047037 13482 raft_consensus.cc:702] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:42.047072 13482 consensus_queue.cc:231] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:42.047255 13482 ts_tablet_manager.cc:1132] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.000s user 0.003s sys 0.000s I1121 10:21:42.047458 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-69c25581de6e4945a167245a4dcbbfe0 received from {username='slave'} at 127.5.70.6:37621 I1121 10:21:42.047492 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-69c25581de6e4945a167245a4dcbbfe0 on tablet 69c25581de6e4945a167245a4dcbbfe0 with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:42.060257 12265 raft_consensus.cc:1123] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.061808 12389 raft_consensus.cc:1123] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.075554 13490 tablet_copy_client.cc:420] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:42.076341 13490 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:42.079808 13490 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:42.079972 13490 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:42.080049 13490 ts_tablet_manager.cc:1104] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s I1121 10:21:42.080178 13490 raft_consensus.cc:337] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:42.080197 13490 raft_consensus.cc:702] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:42.080231 13490 consensus_queue.cc:231] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:42.080322 13490 ts_tablet_manager.cc:1132] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:42.080729 10845 tablet_copy_service.cc:325] P cd693d36f0014dc3a7c3f26196124794: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-702635e315b940138670eeb5ae6ace1b received from {username='slave'} at 127.5.70.8:48773 I1121 10:21:42.080757 10845 tablet_copy_service.cc:413] P cd693d36f0014dc3a7c3f26196124794: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-702635e315b940138670eeb5ae6ace1b on tablet 702635e315b940138670eeb5ae6ace1b with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:42.101660 12389 raft_consensus.cc:1123] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.111848 13606 tablet_copy_client.cc:420] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:42.112617 13606 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:42.115746 13606 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:42.115916 13606 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:42.115986 13606 ts_tablet_manager.cc:1104] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:21:42.116106 13606 raft_consensus.cc:337] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:42.116129 13606 raft_consensus.cc:702] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:42.116178 13606 consensus_queue.cc:231] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:42.116267 13606 ts_tablet_manager.cc:1132] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:42.116493 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-7b34e50304ff46e59622de2e979ac7b8 received from {username='slave'} at 127.5.70.7:54065 I1121 10:21:42.116554 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-7b34e50304ff46e59622de2e979ac7b8 on tablet 7b34e50304ff46e59622de2e979ac7b8 with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:42.121280 12513 raft_consensus.cc:1123] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.172329 13628 tablet_copy_client.cc:420] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:42.173151 13628 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:42.176470 13628 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:42.176712 13628 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:42.176806 13628 ts_tablet_manager.cc:1104] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s I1121 10:21:42.176975 13628 raft_consensus.cc:337] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:42.176995 13628 raft_consensus.cc:702] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:42.177031 13628 consensus_queue.cc:231] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:42.177107 13628 ts_tablet_manager.cc:1132] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:42.177397 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-fa85115a8bbd45dbb7ce1f18b923022a received from {username='slave'} at 127.5.70.5:36527 I1121 10:21:42.177428 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-fa85115a8bbd45dbb7ce1f18b923022a on tablet fa85115a8bbd45dbb7ce1f18b923022a with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:42.230790 12265 raft_consensus.cc:1123] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.265839 12265 raft_consensus.cc:1123] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.298965 12265 raft_consensus.cc:1123] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.323173 12637 raft_consensus.cc:1123] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.331056 12389 raft_consensus.cc:1123] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.368750 12637 raft_consensus.cc:1123] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.390655 13709 raft_consensus.cc:971] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:42.390921 13709 consensus_queue.cc:208] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:42.391381 12513 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.391436 10815 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.391530 11205 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.391667 13709 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.391767 14345 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.391942 14343 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.392849 13709 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.392915 10815 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.393283 12513 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.394032 11205 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.394512 10701 catalog_manager.cc:4143] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:42.395743 11080 consensus_queue.cc:208] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:42.396104 12513 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.396152 10815 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.396338 14343 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.396399 14345 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.397090 14346 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.397315 10815 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.397789 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet a40d6cf45df24788adc85f238c9bcc7e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.398016 12513 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.398393 10707 catalog_manager.cc:4143] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:42.399250 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet a40d6cf45df24788adc85f238c9bcc7e with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:42.399494 14354 tablet_replica.cc:282] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:42.399565 14354 raft_consensus.cc:2156] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.399585 14354 raft_consensus.cc:2183] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.399929 14354 ts_tablet_manager.cc:1457] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.401435 14354 ts_tablet_manager.cc:1470] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.401479 14354 log.cc:1153] T a40d6cf45df24788adc85f238c9bcc7e P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/a40d6cf45df24788adc85f238c9bcc7e I1121 10:21:42.401876 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet a40d6cf45df24788adc85f238c9bcc7e (table rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]) successfully deleted I1121 10:21:42.404989 12265 raft_consensus.cc:1123] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.416878 13709 raft_consensus.cc:971] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:42.417062 13709 consensus_queue.cc:208] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.417505 10815 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.417520 12389 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.417529 10954 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.417735 13709 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.417840 14343 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.417920 14345 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.418866 14345 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.419143 12389 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.419173 10815 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.419554 12637 raft_consensus.cc:1123] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.420370 10954 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.420605 10709 catalog_manager.cc:4143] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.422057 11080 consensus_queue.cc:208] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.422540 10954 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.422592 12389 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.422744 13709 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.422857 14345 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.423838 13709 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.423947 12389 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.424851 10709 catalog_manager.cc:4143] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.425606 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:42.425766 14359 tablet_replica.cc:282] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:42.425850 14359 raft_consensus.cc:2156] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.425879 14359 raft_consensus.cc:2183] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.426111 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.426180 10954 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.426239 14359 ts_tablet_manager.cc:1457] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.428122 14359 ts_tablet_manager.cc:1470] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.428162 14359 log.cc:1153] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/6765d5e95b8f45d4a8f0bb9e5d76b9f0 I1121 10:21:42.428562 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 (table rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]) successfully deleted I1121 10:21:42.444097 14343 raft_consensus.cc:971] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:42.444303 14343 consensus_queue.cc:208] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:42.444901 10954 raft_consensus.cc:1181] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.444937 12513 raft_consensus.cc:1181] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.444962 10815 raft_consensus.cc:1181] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.445123 14346 consensus_queue.cc:984] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.445199 14343 consensus_queue.cc:984] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.445287 13709 consensus_queue.cc:984] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.446099 14346 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.446276 10815 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.446444 12513 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.447396 10954 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.447592 10707 catalog_manager.cc:4143] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce reported cstate change: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.448710 11080 consensus_queue.cc:208] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:42.449085 12513 raft_consensus.cc:1181] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.449157 10815 raft_consensus.cc:1181] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.449290 14345 consensus_queue.cc:984] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.449367 13709 consensus_queue.cc:984] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.449923 13709 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.450075 10815 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.450222 12513 raft_consensus.cc:2801] T 204d260bb58b494091ae91cb57a88ca8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.450728 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 204d260bb58b494091ae91cb57a88ca8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.450989 10709 catalog_manager.cc:4143] T 204d260bb58b494091ae91cb57a88ca8 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.451721 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 204d260bb58b494091ae91cb57a88ca8 with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:42.451892 14364 tablet_replica.cc:282] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:42.451982 14364 raft_consensus.cc:2156] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.452020 14364 raft_consensus.cc:2183] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.452342 14364 ts_tablet_manager.cc:1457] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.453145 14346 raft_consensus.cc:971] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:42.453299 14346 consensus_queue.cc:208] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.453740 10815 raft_consensus.cc:1181] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.453847 12265 raft_consensus.cc:1181] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.453932 14345 consensus_queue.cc:984] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.454026 14364 ts_tablet_manager.cc:1470] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.454046 13709 consensus_queue.cc:984] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.454119 14364 log.cc:1153] T 204d260bb58b494091ae91cb57a88ca8 P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/204d260bb58b494091ae91cb57a88ca8 I1121 10:21:42.454581 10954 raft_consensus.cc:1181] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.454671 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 204d260bb58b494091ae91cb57a88ca8 (table rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]) successfully deleted I1121 10:21:42.454902 13709 consensus_queue.cc:984] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.454943 14345 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.455157 12265 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.455246 10954 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.455941 10815 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.456113 10707 catalog_manager.cc:4143] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.456975 11080 consensus_queue.cc:208] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.457382 10815 raft_consensus.cc:1181] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.457409 12265 raft_consensus.cc:1181] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.457562 14345 consensus_queue.cc:984] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.457626 14346 consensus_queue.cc:984] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.458212 14343 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.458369 12265 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.458856 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 51d8830759cc4842af19cc0d22bb8d65 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.459038 10709 catalog_manager.cc:4143] T 51d8830759cc4842af19cc0d22bb8d65 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:42.459101 10815 raft_consensus.cc:2801] T 51d8830759cc4842af19cc0d22bb8d65 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.459439 12513 raft_consensus.cc:1123] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.459707 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 51d8830759cc4842af19cc0d22bb8d65 with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:42.460103 14364 tablet_replica.cc:282] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:42.460156 14364 raft_consensus.cc:2156] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.460178 14364 raft_consensus.cc:2183] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.460393 14364 ts_tablet_manager.cc:1457] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.461697 14364 ts_tablet_manager.cc:1470] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.461730 14364 log.cc:1153] T 51d8830759cc4842af19cc0d22bb8d65 P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/51d8830759cc4842af19cc0d22bb8d65 I1121 10:21:42.462062 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 51d8830759cc4842af19cc0d22bb8d65 (table rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]) successfully deleted I1121 10:21:42.487838 13788 raft_consensus.cc:971] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:42.488101 13788 consensus_queue.cc:208] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.488617 12389 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.488679 11205 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.488855 10815 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.488883 14369 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.488952 13788 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.489035 13788 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.490010 14372 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.490110 11205 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.490984 10815 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.491245 10709 catalog_manager.cc:4143] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.491305 12637 raft_consensus.cc:1123] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.491389 12389 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.493659 10954 consensus_queue.cc:208] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.494071 12389 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.494072 10815 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) W1121 10:21:42.494212 10692 catalog_manager.cc:3695] ChangeConfig:REMOVE_PEER RPC for tablet 114dbe156d634fca957db3164f8afabb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } }. Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.494284 13788 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.494515 14372 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.494926 13788 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.495077 12389 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.495393 10815 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.495941 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 114dbe156d634fca957db3164f8afabb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.496067 10705 catalog_manager.cc:4143] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.497000 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 114dbe156d634fca957db3164f8afabb with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:42.497069 14354 tablet_replica.cc:282] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:42.497119 14354 raft_consensus.cc:2156] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.497134 14354 raft_consensus.cc:2183] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.497467 14354 ts_tablet_manager.cc:1457] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.498912 14354 ts_tablet_manager.cc:1470] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.498944 14354 log.cc:1153] T 114dbe156d634fca957db3164f8afabb P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/114dbe156d634fca957db3164f8afabb I1121 10:21:42.499313 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 114dbe156d634fca957db3164f8afabb (table rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]) successfully deleted I1121 10:21:42.526571 10692 catalog_manager.cc:3712] ChangeConfig:REMOVE_PEER RPC for tablet 114dbe156d634fca957db3164f8afabb with cas_config_opid_index 5: aborting the task: latest config opid_index 6; task opid_index 5 I1121 10:21:42.527050 12389 raft_consensus.cc:1123] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:42.528232 14369 raft_consensus.cc:971] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:42.528420 14369 consensus_queue.cc:208] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.528928 11205 raft_consensus.cc:1181] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.528934 12389 raft_consensus.cc:1181] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.529130 14372 consensus_queue.cc:984] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.529222 13788 consensus_queue.cc:984] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.529611 11080 raft_consensus.cc:1181] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.529844 14372 consensus_queue.cc:984] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.530354 14371 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.530670 12389 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.530963 11080 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.531821 10701 catalog_manager.cc:4143] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:42.532255 11205 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.533040 10953 consensus_queue.cc:208] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.533495 12389 raft_consensus.cc:1181] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.533723 13788 consensus_queue.cc:984] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.533816 11205 raft_consensus.cc:1181] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.534081 14372 consensus_queue.cc:984] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.534445 13788 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.534786 11205 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.535332 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 0ec67ec228044beda81b53a3e842af49 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.535604 12389 raft_consensus.cc:2801] T 0ec67ec228044beda81b53a3e842af49 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.535796 10707 catalog_manager.cc:4143] T 0ec67ec228044beda81b53a3e842af49 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.537806 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 0ec67ec228044beda81b53a3e842af49 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:42.537997 14385 tablet_replica.cc:282] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:42.538230 14385 raft_consensus.cc:2156] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.538288 14385 raft_consensus.cc:2183] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.538664 14385 ts_tablet_manager.cc:1457] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.540424 14385 ts_tablet_manager.cc:1470] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.540555 14385 log.cc:1153] T 0ec67ec228044beda81b53a3e842af49 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/0ec67ec228044beda81b53a3e842af49 I1121 10:21:42.541024 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 0ec67ec228044beda81b53a3e842af49 (table rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]) successfully deleted I1121 10:21:42.574862 14016 raft_consensus.cc:971] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:42.575143 14016 consensus_queue.cc:208] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:42.575703 12637 raft_consensus.cc:1181] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.575702 10953 raft_consensus.cc:1181] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.575726 11080 raft_consensus.cc:1181] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.575929 14426 consensus_queue.cc:984] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.576017 14016 consensus_queue.cc:984] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.576136 14426 consensus_queue.cc:984] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.577265 14426 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.577368 11080 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.577528 10953 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.578527 10701 catalog_manager.cc:4143] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.578783 12637 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.579416 11205 consensus_queue.cc:208] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:42.579826 11080 raft_consensus.cc:1181] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.579902 12637 raft_consensus.cc:1181] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.580070 14443 consensus_queue.cc:984] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.580121 14016 consensus_queue.cc:984] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.580741 14016 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.580871 11080 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.580945 12637 raft_consensus.cc:2801] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.581858 10707 catalog_manager.cc:4143] T 4ef99b04fd634cd599c435ad3383c4d5 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.582144 10693 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 4ef99b04fd634cd599c435ad3383c4d5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.582890 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 4ef99b04fd634cd599c435ad3383c4d5 with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:42.582988 14364 tablet_replica.cc:282] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:42.583039 14364 raft_consensus.cc:2156] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.583065 14364 raft_consensus.cc:2183] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.583300 14364 ts_tablet_manager.cc:1457] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.584911 14364 ts_tablet_manager.cc:1470] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.584957 14364 log.cc:1153] T 4ef99b04fd634cd599c435ad3383c4d5 P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/4ef99b04fd634cd599c435ad3383c4d5 I1121 10:21:42.585515 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 4ef99b04fd634cd599c435ad3383c4d5 (table rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]) successfully deleted I1121 10:21:42.609841 11205 consensus_queue.cc:208] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:42.610420 10815 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:42.610448 11080 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:42.610464 11136 consensus_peers.cc:473] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fa5ac03774f4944815d262fcdffa2f6. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:42.610646 14016 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.610733 14443 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.611523 14016 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.611651 10815 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.611699 11080 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.612709 10701 catalog_manager.cc:4143] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:42.617367 10815 consensus_queue.cc:208] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:42.617893 11080 raft_consensus.cc:1181] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:42.618028 11205 raft_consensus.cc:1181] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:42.618242 14502 consensus_queue.cc:984] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.618425 14503 consensus_queue.cc:984] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.619253 14502 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:42.619534 11080 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:42.619581 11205 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:42.620749 10707 catalog_manager.cc:4143] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W1121 10:21:42.621861 10763 consensus_peers.cc:473] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc935fc8b0984388a5db91f6e37824fa. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:42.624933 11205 consensus_queue.cc:208] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:42.625408 10815 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:42.625452 10953 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:42.625460 11136 consensus_peers.cc:473] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 172c695aadee4de6ad0a5a6508b290f6. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:42.625597 14443 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.625666 14444 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.626540 14443 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.626708 10953 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.626768 10815 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.627831 10705 catalog_manager.cc:4143] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:42.632055 11080 consensus_queue.cc:208] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:42.632252 14372 raft_consensus.cc:971] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:42.632422 14372 consensus_queue.cc:208] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:42.632583 10815 raft_consensus.cc:1181] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:42.632791 11011 consensus_peers.cc:473] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dfbf2f060fb481fb74acc019b6eaebc. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:42.632820 11205 raft_consensus.cc:1181] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:42.632879 14346 consensus_queue.cc:984] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.633000 11080 raft_consensus.cc:1181] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.633040 14345 consensus_queue.cc:984] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.633201 14369 consensus_queue.cc:984] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.633448 12513 raft_consensus.cc:1181] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.633646 14369 consensus_queue.cc:984] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.633662 10815 raft_consensus.cc:1181] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.633689 14346 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:42.633886 14369 consensus_queue.cc:984] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.634089 11205 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:42.634536 14371 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.634673 10815 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.635280 10701 catalog_manager.cc:4143] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:42.635630 12513 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.635818 11080 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.636611 10815 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:42.636912 10705 catalog_manager.cc:4143] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.638044 10953 consensus_queue.cc:208] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:42.638490 12513 raft_consensus.cc:1181] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.638677 14372 consensus_queue.cc:984] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.638712 10815 raft_consensus.cc:1181] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.638957 14372 consensus_queue.cc:984] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.639322 14372 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.639726 10815 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.640159 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 5becd9fee1eb40cf8eebb4ffac223e15 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.640362 10707 catalog_manager.cc:4143] T 5becd9fee1eb40cf8eebb4ffac223e15 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:42.640373 12513 raft_consensus.cc:2801] T 5becd9fee1eb40cf8eebb4ffac223e15 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.641023 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 5becd9fee1eb40cf8eebb4ffac223e15 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:42.641299 14385 tablet_replica.cc:282] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:42.641352 14385 raft_consensus.cc:2156] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.641369 14385 raft_consensus.cc:2183] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.641705 14385 ts_tablet_manager.cc:1457] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.643007 14385 ts_tablet_manager.cc:1470] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.643052 14385 log.cc:1153] T 5becd9fee1eb40cf8eebb4ffac223e15 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/5becd9fee1eb40cf8eebb4ffac223e15 I1121 10:21:42.643405 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 5becd9fee1eb40cf8eebb4ffac223e15 (table rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]) successfully deleted I1121 10:21:42.644119 10953 consensus_queue.cc:208] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:42.644563 10815 raft_consensus.cc:1181] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:42.644593 11205 raft_consensus.cc:1181] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:42.644701 10888 consensus_peers.cc:473] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bbbf8a8d2df4f76aa83b5fe842b29a6. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:42.644847 14369 consensus_queue.cc:984] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.644914 14371 consensus_queue.cc:984] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.645687 13788 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.645841 11205 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.646035 10815 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.646775 10707 catalog_manager.cc:4143] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:42.647953 14503 raft_consensus.cc:971] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:42.648161 14503 consensus_queue.cc:208] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.648674 12265 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.648733 10953 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.648746 11080 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.648928 14502 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.648996 14503 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.649082 13711 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.649859 14503 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.650048 10953 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.650053 11080 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.650982 10705 catalog_manager.cc:4143] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.651284 12265 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.652354 10815 consensus_queue.cc:208] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.652892 12265 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.653085 13711 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.653249 10953 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.653424 13711 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.653636 14545 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.653813 12265 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.654086 10953 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.654424 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 40e235c631e14bf8b4fe83e2953d8c87 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.654722 10701 catalog_manager.cc:4143] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 reported cstate change: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.655382 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 40e235c631e14bf8b4fe83e2953d8c87 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:42.655467 14385 tablet_replica.cc:282] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:42.655520 14385 raft_consensus.cc:2156] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.655542 14385 raft_consensus.cc:2183] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.655810 14385 ts_tablet_manager.cc:1457] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.657080 14385 ts_tablet_manager.cc:1470] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.657122 14385 log.cc:1153] T 40e235c631e14bf8b4fe83e2953d8c87 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/40e235c631e14bf8b4fe83e2953d8c87 I1121 10:21:42.657513 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 40e235c631e14bf8b4fe83e2953d8c87 (table rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]) successfully deleted I1121 10:21:42.684068 13709 raft_consensus.cc:971] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:42.684365 13709 consensus_queue.cc:208] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:42.684856 10953 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.684861 12513 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.684866 11205 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.685092 14346 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.685158 13709 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.685240 14343 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.686271 13709 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.686411 12513 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.686537 10953 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.687418 11205 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.687475 10707 catalog_manager.cc:4143] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce reported cstate change: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.688786 11080 consensus_queue.cc:208] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:42.689155 12513 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.689206 11205 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.689409 13709 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.689572 13709 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.690021 14345 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.690158 12513 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.690215 11205 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:42.690732 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 7b34e50304ff46e59622de2e979ac7b8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.691145 10705 catalog_manager.cc:4143] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:42.691934 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 7b34e50304ff46e59622de2e979ac7b8 with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:42.691998 14364 tablet_replica.cc:282] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:42.692046 14364 raft_consensus.cc:2156] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.692067 14364 raft_consensus.cc:2183] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.692325 14364 ts_tablet_manager.cc:1457] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.693953 14364 ts_tablet_manager.cc:1470] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.693996 14364 log.cc:1153] T 7b34e50304ff46e59622de2e979ac7b8 P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/7b34e50304ff46e59622de2e979ac7b8 I1121 10:21:42.694422 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 7b34e50304ff46e59622de2e979ac7b8 (table rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]) successfully deleted I1121 10:21:42.715952 10690 catalog_manager.cc:3712] ChangeConfig:REMOVE_PEER RPC for tablet 40e235c631e14bf8b4fe83e2953d8c87 with cas_config_opid_index 5: aborting the task: latest config opid_index 6; task opid_index 5 I1121 10:21:42.719877 11205 consensus_queue.cc:208] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:42.720362 10953 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:42.720427 10816 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:42.720466 11136 consensus_peers.cc:473] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df2330a22c8142ada6d056401d0a064d. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:42.720680 14016 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.720772 14443 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.721581 14426 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:42.721719 10816 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:42.721793 10953 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:42.723069 10701 catalog_manager.cc:4143] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:42.727407 11205 consensus_queue.cc:208] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:42.728077 10816 raft_consensus.cc:1181] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:42.728090 11080 raft_consensus.cc:1181] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:42.728297 14443 consensus_queue.cc:984] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.728384 14444 consensus_queue.cc:984] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:42.729176 14443 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:42.729322 11080 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:42.729964 10816 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:42.730515 10705 catalog_manager.cc:4143] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W1121 10:21:42.731271 11136 consensus_peers.cc:473] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e5ec5084c6a4d6089c2cb198367d342. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:42.746399 14503 raft_consensus.cc:971] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:42.746596 14503 consensus_queue.cc:208] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.747002 11205 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.747035 10953 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.747054 12265 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.747220 13711 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.747298 14503 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.747653 14502 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.748198 14545 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.748760 10953 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.749375 12265 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.749799 11205 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.750475 10709 catalog_manager.cc:4143] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.751941 10816 consensus_queue.cc:208] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.752269 12265 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.752310 11205 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.752540 14503 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.752638 14502 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.753186 14503 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.753325 11205 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.753995 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet c7de6e746a8246f89d8b65b59803c3af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.754098 14369 raft_consensus.cc:971] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:42.754238 12265 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.754253 14369 consensus_queue.cc:208] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.754597 10705 catalog_manager.cc:4143] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:42.754644 11205 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.754828 13788 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.754824 12389 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.755017 13788 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.755179 11080 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.755314 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet c7de6e746a8246f89d8b65b59803c3af with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:42.755385 14364 tablet_replica.cc:282] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:42.755445 14364 raft_consensus.cc:2156] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.755471 14364 raft_consensus.cc:2183] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.755712 13788 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.755725 14364 ts_tablet_manager.cc:1457] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.757198 14364 ts_tablet_manager.cc:1470] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.757364 14364 log.cc:1153] T c7de6e746a8246f89d8b65b59803c3af P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/c7de6e746a8246f89d8b65b59803c3af I1121 10:21:42.757859 13788 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.757885 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet c7de6e746a8246f89d8b65b59803c3af (table rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]) successfully deleted I1121 10:21:42.758183 12389 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.759088 11205 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.759106 11080 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.760089 10701 catalog_manager.cc:4143] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.761108 10953 consensus_queue.cc:208] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.761447 12389 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.761472 11205 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.761679 14369 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.761813 14372 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.762416 14369 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.762521 12389 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.762578 11205 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.763353 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 69c25581de6e4945a167245a4dcbbfe0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.763509 10707 catalog_manager.cc:4143] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.764214 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 69c25581de6e4945a167245a4dcbbfe0 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:42.764276 14385 tablet_replica.cc:282] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:42.764325 14385 raft_consensus.cc:2156] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.764343 14385 raft_consensus.cc:2183] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.764607 14385 ts_tablet_manager.cc:1457] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.766089 14385 ts_tablet_manager.cc:1470] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.766135 14385 log.cc:1153] T 69c25581de6e4945a167245a4dcbbfe0 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/69c25581de6e4945a167245a4dcbbfe0 I1121 10:21:42.766451 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 69c25581de6e4945a167245a4dcbbfe0 (table rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]) successfully deleted I1121 10:21:42.789538 13709 raft_consensus.cc:971] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:42.789806 13709 consensus_queue.cc:208] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.790267 10816 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.790398 12265 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.790431 10953 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.790829 14345 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.790949 14346 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.791450 13709 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.792290 13709 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.792552 10816 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.792605 12265 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.793668 10953 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.793797 10707 catalog_manager.cc:4143] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.794700 11080 consensus_queue.cc:208] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.795104 12265 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.795233 10953 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.795300 14345 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.795397 14345 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.795965 14343 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.796209 10954 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.796739 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet fa85115a8bbd45dbb7ce1f18b923022a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.797346 10705 catalog_manager.cc:4143] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:42.797438 12265 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.798066 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet fa85115a8bbd45dbb7ce1f18b923022a with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:42.798135 14359 tablet_replica.cc:282] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:42.798182 14359 raft_consensus.cc:2156] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.798198 14359 raft_consensus.cc:2183] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.798502 14359 ts_tablet_manager.cc:1457] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.800113 14359 ts_tablet_manager.cc:1470] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.800150 14359 log.cc:1153] T fa85115a8bbd45dbb7ce1f18b923022a P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/fa85115a8bbd45dbb7ce1f18b923022a I1121 10:21:42.800662 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet fa85115a8bbd45dbb7ce1f18b923022a (table rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]) successfully deleted I1121 10:21:42.805052 14502 raft_consensus.cc:971] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:42.805240 14502 consensus_queue.cc:208] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:42.805672 11080 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.805719 10954 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.805892 12637 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.805960 13711 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.806035 14503 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.806154 14502 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.807041 14503 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.807181 12637 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.807454 11080 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.808151 10954 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.808288 10705 catalog_manager.cc:4143] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.809396 10816 consensus_queue.cc:208] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:42.809751 10954 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.809931 14502 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.809957 12637 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.810178 14502 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.810747 13711 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.810889 10954 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.810940 12637 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.811689 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 3e0bc1c59a1b4e2c9e761a6de8085226 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.811713 10709 catalog_manager.cc:4143] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.812469 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 3e0bc1c59a1b4e2c9e761a6de8085226 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:42.812602 14385 tablet_replica.cc:282] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:42.812656 14385 raft_consensus.cc:2156] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.812677 14385 raft_consensus.cc:2183] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.813063 14385 ts_tablet_manager.cc:1457] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.814558 14385 ts_tablet_manager.cc:1470] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.814599 14385 log.cc:1153] T 3e0bc1c59a1b4e2c9e761a6de8085226 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/3e0bc1c59a1b4e2c9e761a6de8085226 I1121 10:21:42.815016 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 3e0bc1c59a1b4e2c9e761a6de8085226 (table rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]) successfully deleted I1121 10:21:42.921999 14346 raft_consensus.cc:971] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:42.922211 14346 consensus_queue.cc:208] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:42.922713 12637 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.922842 11202 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.922982 10816 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.923003 14345 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.923079 14343 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.923207 14343 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.924239 13709 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.924407 10816 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.924410 11202 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.925498 12637 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.925550 10705 catalog_manager.cc:4143] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.927002 11080 consensus_queue.cc:208] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:42.927466 11202 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.927474 12637 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.927665 14346 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.927739 14343 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.928282 14346 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.928406 11202 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.928671 12637 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.929205 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 8963dd2bc2da4cacb9d5e5329680d7d7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.929430 10709 catalog_manager.cc:4143] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.930092 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet 8963dd2bc2da4cacb9d5e5329680d7d7 with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:42.930160 14359 tablet_replica.cc:282] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:42.930207 14359 raft_consensus.cc:2156] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.930227 14359 raft_consensus.cc:2183] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.930481 14359 ts_tablet_manager.cc:1457] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.932503 14359 ts_tablet_manager.cc:1470] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.932588 14359 log.cc:1153] T 8963dd2bc2da4cacb9d5e5329680d7d7 P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/8963dd2bc2da4cacb9d5e5329680d7d7 I1121 10:21:42.932936 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet 8963dd2bc2da4cacb9d5e5329680d7d7 (table rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]) successfully deleted I1121 10:21:42.959547 14502 raft_consensus.cc:971] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:42.959769 14502 consensus_queue.cc:208] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:42.960268 12637 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.960297 11202 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.960378 11080 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.960491 14545 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.960624 13711 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.960840 13711 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.961498 13711 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.961635 11202 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.961745 11080 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.962711 12637 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.962733 10707 catalog_manager.cc:4143] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:42.964006 10816 consensus_queue.cc:208] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:42.964393 11080 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.964453 12637 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.964658 14502 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.964754 14503 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.965344 14503 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.965492 12637 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.965539 11080 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.966132 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 702635e315b940138670eeb5ae6ace1b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.966300 10709 catalog_manager.cc:4143] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:42.967005 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 702635e315b940138670eeb5ae6ace1b with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:42.967080 14354 tablet_replica.cc:282] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:42.967149 14354 raft_consensus.cc:2156] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.967164 14354 raft_consensus.cc:2183] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.967480 14354 ts_tablet_manager.cc:1457] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.968811 14354 ts_tablet_manager.cc:1470] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.968847 14354 log.cc:1153] T 702635e315b940138670eeb5ae6ace1b P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/702635e315b940138670eeb5ae6ace1b I1121 10:21:42.969147 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 702635e315b940138670eeb5ae6ace1b (table rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]) successfully deleted I1121 10:21:42.976141 14372 raft_consensus.cc:971] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:42.976372 14372 consensus_queue.cc:208] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.977035 12265 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.977062 11202 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.977154 10816 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.977289 13788 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.977419 14372 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.977483 14371 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.978483 14372 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.978592 11202 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.979480 12265 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.979696 10816 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.980644 10707 catalog_manager.cc:4143] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 reported cstate change: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.981173 14443 raft_consensus.cc:971] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:42.981361 14443 consensus_queue.cc:208] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.981844 10954 consensus_queue.cc:208] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:42.981910 12389 raft_consensus.cc:1181] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.982009 10953 raft_consensus.cc:1181] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.982138 14426 consensus_queue.cc:984] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.982195 10816 raft_consensus.cc:1181] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:42.982324 14444 consensus_queue.cc:984] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.982412 14444 consensus_queue.cc:984] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:42.982630 12265 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.982830 13788 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.982875 10815 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.983048 13788 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.983495 14371 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.983760 14444 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.983829 10816 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.983876 10953 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.984474 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 25b55ff241d8442897ea97523a2cd6fd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.984879 12389 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.984935 10701 catalog_manager.cc:4143] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.984997 12265 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:42.985610 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 25b55ff241d8442897ea97523a2cd6fd with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:42.985678 14354 tablet_replica.cc:282] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:42.985733 14354 raft_consensus.cc:2156] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.985765 14354 raft_consensus.cc:2183] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.985882 10701 catalog_manager.cc:4143] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.986274 10816 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.986814 11202 consensus_queue.cc:208] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:42.986935 14354 ts_tablet_manager.cc:1457] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.987228 12389 raft_consensus.cc:1181] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.987424 14443 consensus_queue.cc:984] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.988075 10953 raft_consensus.cc:1181] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:42.988293 14016 consensus_queue.cc:984] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:42.988322 14443 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.988404 14354 ts_tablet_manager.cc:1470] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.988440 14354 log.cc:1153] T 25b55ff241d8442897ea97523a2cd6fd P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/25b55ff241d8442897ea97523a2cd6fd I1121 10:21:42.988644 10953 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.988849 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 25b55ff241d8442897ea97523a2cd6fd (table rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]) successfully deleted I1121 10:21:42.989099 10693 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet cc68d4b786cd485285707592d3c25f80 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:42.989261 12389 raft_consensus.cc:2801] T cc68d4b786cd485285707592d3c25f80 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:42.989747 10709 catalog_manager.cc:4143] T cc68d4b786cd485285707592d3c25f80 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:42.989987 14914 ts_tablet_manager.cc:721] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:42.990068 14914 tablet_copy_client.cc:243] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:37207 I1121 10:21:42.990232 11218 tablet_copy_service.cc:135] P 5457a73f9ecb40ee9889513a8412937b: Received BeginTabletCopySession request for tablet 1fa5ac03774f4944815d262fcdffa2f6 from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:39461) I1121 10:21:42.990259 11218 tablet_copy_service.cc:156] P 5457a73f9ecb40ee9889513a8412937b: Beginning new tablet copy session on tablet 1fa5ac03774f4944815d262fcdffa2f6 from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:39461: session id = 2c3332f15c014e68bed03f69d8c6287d-1fa5ac03774f4944815d262fcdffa2f6 I1121 10:21:42.990382 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet cc68d4b786cd485285707592d3c25f80 with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:42.990435 14359 tablet_replica.cc:282] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:42.990474 14359 raft_consensus.cc:2156] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:42.990487 14359 raft_consensus.cc:2183] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:42.990753 14359 ts_tablet_manager.cc:1457] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:42.990809 11218 tablet_copy_source_session.cc:204] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:42.991164 14914 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 1fa5ac03774f4944815d262fcdffa2f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:42.992027 14914 tablet_copy_client.cc:581] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:42.992099 14359 ts_tablet_manager.cc:1470] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:42.992136 14359 log.cc:1153] T cc68d4b786cd485285707592d3c25f80 P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/cc68d4b786cd485285707592d3c25f80 I1121 10:21:42.992259 14914 tablet_copy_client.cc:544] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:42.992566 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet cc68d4b786cd485285707592d3c25f80 (table rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]) successfully deleted I1121 10:21:42.993073 14914 tablet_copy_client.cc:755] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 I1121 10:21:43.013970 14372 raft_consensus.cc:971] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:43.014189 14372 consensus_queue.cc:208] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:43.014637 12513 raft_consensus.cc:1181] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:43.014643 11080 raft_consensus.cc:1181] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:43.014845 11202 raft_consensus.cc:1181] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:43.014883 14371 consensus_queue.cc:984] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:43.014931 14369 consensus_queue.cc:984] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:43.015080 13788 consensus_queue.cc:984] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:43.016052 14371 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:43.016211 12513 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:43.016340 11080 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:43.017572 10701 catalog_manager.cc:4143] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce reported cstate change: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:43.017688 11202 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:43.018532 10953 consensus_queue.cc:208] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:43.018869 12513 raft_consensus.cc:1181] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:43.018910 11080 raft_consensus.cc:1181] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:43.019064 13788 consensus_queue.cc:984] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:43.019132 14372 consensus_queue.cc:984] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:43.019776 14372 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:43.019937 12513 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:43.020066 11080 raft_consensus.cc:2801] T 45f0a219c25243fbade25321fcc1067a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:43.020606 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 45f0a219c25243fbade25321fcc1067a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:43.021026 10709 catalog_manager.cc:4143] T 45f0a219c25243fbade25321fcc1067a P 77830a7609a549e0a1c76586d0e301ce reported cstate change: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:43.021647 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 45f0a219c25243fbade25321fcc1067a with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:43.021749 14354 tablet_replica.cc:282] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:43.021808 14354 raft_consensus.cc:2156] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:43.021832 14354 raft_consensus.cc:2183] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:43.022110 14354 ts_tablet_manager.cc:1457] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:43.023689 14354 ts_tablet_manager.cc:1470] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:43.023726 14354 log.cc:1153] T 45f0a219c25243fbade25321fcc1067a P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/45f0a219c25243fbade25321fcc1067a I1121 10:21:43.024160 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 45f0a219c25243fbade25321fcc1067a (table rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]) successfully deleted I1121 10:21:43.105238 14371 raft_consensus.cc:971] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:43.105453 14371 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:43.105901 10816 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:43.105967 11080 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:43.105980 12637 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:43.106146 14372 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:43.106251 14372 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:43.106324 14369 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:43.107283 14369 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:43.107504 11080 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:43.107726 12637 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:43.108656 10701 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:43.108705 10816 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:43.109441 10953 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:43.109890 12637 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:43.109995 11080 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:43.110378 14369 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:43.110483 14371 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:43.111068 14371 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:43.111346 11080 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:43.111867 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 4ecac46041da45aba6e82df02354fe89 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:43.112260 10705 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:43.113029 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet 4ecac46041da45aba6e82df02354fe89 with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:43.113111 14359 tablet_replica.cc:282] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:43.113158 14359 raft_consensus.cc:2156] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:43.113174 14359 raft_consensus.cc:2183] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:43.113487 14359 ts_tablet_manager.cc:1457] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:43.113833 12637 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:43.115314 14359 ts_tablet_manager.cc:1470] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:43.115352 14359 log.cc:1153] T 4ecac46041da45aba6e82df02354fe89 P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/4ecac46041da45aba6e82df02354fe89 I1121 10:21:43.115682 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet 4ecac46041da45aba6e82df02354fe89 (table rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]) successfully deleted I1121 10:21:43.125562 14963 ts_tablet_manager.cc:721] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:43.125622 14963 tablet_copy_client.cc:243] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:43035 I1121 10:21:43.128716 10845 tablet_copy_service.cc:135] P cd693d36f0014dc3a7c3f26196124794: Received BeginTabletCopySession request for tablet cc935fc8b0984388a5db91f6e37824fa from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:53947) I1121 10:21:43.128753 10845 tablet_copy_service.cc:156] P cd693d36f0014dc3a7c3f26196124794: Beginning new tablet copy session on tablet cc935fc8b0984388a5db91f6e37824fa from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:53947: session id = 77830a7609a549e0a1c76586d0e301ce-cc935fc8b0984388a5db91f6e37824fa I1121 10:21:43.129338 10845 tablet_copy_source_session.cc:204] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:43.129695 14963 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet cc935fc8b0984388a5db91f6e37824fa. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:43.131090 14963 tablet_copy_client.cc:581] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:43.131296 14963 tablet_copy_client.cc:544] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:43.131791 14963 tablet_copy_client.cc:755] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:43.134248 14980 ts_tablet_manager.cc:721] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:43.134289 14980 tablet_copy_client.cc:243] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:43.134511 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 4dfbf2f060fb481fb74acc019b6eaebc from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:36527) I1121 10:21:43.134548 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 4dfbf2f060fb481fb74acc019b6eaebc from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:36527: session id = fed6a935d10042b48cdf1f30824f6c57-4dfbf2f060fb481fb74acc019b6eaebc I1121 10:21:43.135119 11085 tablet_copy_source_session.cc:204] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:43.135572 14980 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4dfbf2f060fb481fb74acc019b6eaebc. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:43.137122 14980 tablet_copy_client.cc:581] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:43.137315 14980 tablet_copy_client.cc:544] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:43.137794 14980 tablet_copy_client.cc:755] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:43.146462 11202 consensus_queue.cc:208] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:43.146931 10953 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:43.147055 11080 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:43.147171 14444 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:43.147307 14426 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:43.148043 14444 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:43.148260 10953 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:43.148326 11080 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:43.149586 10709 catalog_manager.cc:4143] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index -1 to 4, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } W1121 10:21:43.150552 11135 consensus_peers.cc:473] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f7d9348eaaa4477b9dd581c28e2d5e3. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:43.153582 11080 consensus_queue.cc:208] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:43.154050 11202 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:43.154103 10816 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:21:43.154193 11012 consensus_peers.cc:473] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69ac58ad89e447a494fc407037d27f8d. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:43.154322 14345 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:43.154405 13709 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:43.155305 14343 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:43.155477 11202 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:43.155586 10816 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:43.156442 10705 catalog_manager.cc:4143] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index -1 to 4, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:43.157373 15008 ts_tablet_manager.cc:721] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:43.157414 15008 tablet_copy_client.cc:243] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:43.157600 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:37621) I1121 10:21:43.157649 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:37621: session id = 2c3332f15c014e68bed03f69d8c6287d-8bbbf8a8d2df4f76aa83b5fe842b29a6 I1121 10:21:43.158248 10977 tablet_copy_source_session.cc:204] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:43.158601 15008 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:43.160182 15008 tablet_copy_client.cc:581] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:43.160284 15008 tablet_copy_client.cc:544] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:43.160872 15008 tablet_copy_client.cc:755] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 I1121 10:21:43.165267 11080 consensus_queue.cc:208] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:43.165872 10953 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:43.166049 11202 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:21:43.166061 14345 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s W1121 10:21:43.166108 11012 consensus_peers.cc:473] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 635b29a1f41e418faffe3677886048e5. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:43.166353 14345 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:21:43.167102 14346 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:43.167291 10953 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:43.167665 11202 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:43.168579 10709 catalog_manager.cc:4143] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index -1 to 4, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:43.232770 15067 ts_tablet_manager.cc:721] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:43.232836 15067 tablet_copy_client.cc:243] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:37207 I1121 10:21:43.233064 11218 tablet_copy_service.cc:135] P 5457a73f9ecb40ee9889513a8412937b: Received BeginTabletCopySession request for tablet 172c695aadee4de6ad0a5a6508b290f6 from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:39461) I1121 10:21:43.233098 11218 tablet_copy_service.cc:156] P 5457a73f9ecb40ee9889513a8412937b: Beginning new tablet copy session on tablet 172c695aadee4de6ad0a5a6508b290f6 from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:39461: session id = 2c3332f15c014e68bed03f69d8c6287d-172c695aadee4de6ad0a5a6508b290f6 I1121 10:21:43.233557 11218 tablet_copy_source_session.cc:204] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:43.233887 15067 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 172c695aadee4de6ad0a5a6508b290f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:43.234792 15067 tablet_copy_client.cc:581] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:43.234891 15067 tablet_copy_client.cc:544] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:43.235474 15067 tablet_copy_client.cc:755] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 I1121 10:21:43.257427 15068 ts_tablet_manager.cc:721] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:43.257484 15068 tablet_copy_client.cc:243] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:37207 I1121 10:21:43.257699 11218 tablet_copy_service.cc:135] P 5457a73f9ecb40ee9889513a8412937b: Received BeginTabletCopySession request for tablet df2330a22c8142ada6d056401d0a064d from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:35393) I1121 10:21:43.257727 11218 tablet_copy_service.cc:156] P 5457a73f9ecb40ee9889513a8412937b: Beginning new tablet copy session on tablet df2330a22c8142ada6d056401d0a064d from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:35393: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-df2330a22c8142ada6d056401d0a064d I1121 10:21:43.258232 11218 tablet_copy_source_session.cc:204] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:43.258560 15068 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet df2330a22c8142ada6d056401d0a064d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:43.259506 15068 tablet_copy_client.cc:581] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:43.259610 15068 tablet_copy_client.cc:544] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:43.260200 15068 tablet_copy_client.cc:755] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:43.349076 15069 ts_tablet_manager.cc:721] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:43.349128 15069 tablet_copy_client.cc:243] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:37207 I1121 10:21:43.351794 11218 tablet_copy_service.cc:135] P 5457a73f9ecb40ee9889513a8412937b: Received BeginTabletCopySession request for tablet 0e5ec5084c6a4d6089c2cb198367d342 from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:51745) I1121 10:21:43.351824 11218 tablet_copy_service.cc:156] P 5457a73f9ecb40ee9889513a8412937b: Beginning new tablet copy session on tablet 0e5ec5084c6a4d6089c2cb198367d342 from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:51745: session id = 77830a7609a549e0a1c76586d0e301ce-0e5ec5084c6a4d6089c2cb198367d342 I1121 10:21:43.352237 11218 tablet_copy_source_session.cc:204] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:43.352672 15069 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0e5ec5084c6a4d6089c2cb198367d342. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:43.353646 15069 tablet_copy_client.cc:581] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:43.353731 15069 tablet_copy_client.cc:544] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:43.354408 15069 tablet_copy_client.cc:755] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:43.578541 15120 ts_tablet_manager.cc:721] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:43.578591 15120 tablet_copy_client.cc:243] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:43.578984 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 635b29a1f41e418faffe3677886048e5 from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:51069) I1121 10:21:43.579015 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 635b29a1f41e418faffe3677886048e5 from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:51069: session id = 2c3332f15c014e68bed03f69d8c6287d-635b29a1f41e418faffe3677886048e5 I1121 10:21:43.579516 11085 tablet_copy_source_session.cc:204] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:43.580026 15120 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 635b29a1f41e418faffe3677886048e5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:43.580996 15120 tablet_copy_client.cc:581] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:43.581112 15120 tablet_copy_client.cc:544] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:43.581595 15120 tablet_copy_client.cc:755] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 I1121 10:21:43.616698 15122 ts_tablet_manager.cc:721] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:43.616744 15122 tablet_copy_client.cc:243] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:43.616963 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 69ac58ad89e447a494fc407037d27f8d from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:52727) I1121 10:21:43.616991 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 69ac58ad89e447a494fc407037d27f8d from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:52727: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-69ac58ad89e447a494fc407037d27f8d I1121 10:21:43.617445 11085 tablet_copy_source_session.cc:204] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:43.617785 15122 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69ac58ad89e447a494fc407037d27f8d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:43.618742 15122 tablet_copy_client.cc:581] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:43.618844 15122 tablet_copy_client.cc:544] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:43.619406 15122 tablet_copy_client.cc:755] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:43.673509 15173 ts_tablet_manager.cc:721] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:43.673552 15173 tablet_copy_client.cc:243] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:37207 I1121 10:21:43.675870 11218 tablet_copy_service.cc:135] P 5457a73f9ecb40ee9889513a8412937b: Received BeginTabletCopySession request for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:34817) I1121 10:21:43.675902 11218 tablet_copy_service.cc:156] P 5457a73f9ecb40ee9889513a8412937b: Beginning new tablet copy session on tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:34817: session id = fed6a935d10042b48cdf1f30824f6c57-5f7d9348eaaa4477b9dd581c28e2d5e3 I1121 10:21:43.676374 11218 tablet_copy_source_session.cc:204] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:43.676854 15173 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:43.677765 15173 tablet_copy_client.cc:581] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:43.677879 15173 tablet_copy_client.cc:544] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:43.678407 15173 tablet_copy_client.cc:755] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:44.493228 14914 tablet_copy_client.cc:420] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:44.494156 14914 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:44.497408 14914 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:44.497587 14914 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:44.497680 14914 ts_tablet_manager.cc:1104] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I1121 10:21:44.497826 14914 raft_consensus.cc:337] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:44.497854 14914 raft_consensus.cc:702] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:44.498000 14914 consensus_queue.cc:231] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:44.498142 14914 ts_tablet_manager.cc:1132] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:21:44.498464 11218 tablet_copy_service.cc:325] P 5457a73f9ecb40ee9889513a8412937b: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-1fa5ac03774f4944815d262fcdffa2f6 received from {username='slave'} at 127.5.70.6:39461 I1121 10:21:44.498500 11218 tablet_copy_service.cc:413] P 5457a73f9ecb40ee9889513a8412937b: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-1fa5ac03774f4944815d262fcdffa2f6 on tablet 1fa5ac03774f4944815d262fcdffa2f6 with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:44.631937 14963 tablet_copy_client.cc:420] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:44.632694 14963 tablet_bootstrap.cc:439] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:44.635764 14963 tablet_bootstrap.cc:439] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:44.635963 14963 tablet_bootstrap.cc:439] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:44.636059 14963 ts_tablet_manager.cc:1104] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I1121 10:21:44.636199 14963 raft_consensus.cc:337] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:44.636224 14963 raft_consensus.cc:702] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:44.636278 14963 consensus_queue.cc:231] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:44.636394 14963 ts_tablet_manager.cc:1132] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.002s I1121 10:21:44.636646 10845 tablet_copy_service.cc:325] P cd693d36f0014dc3a7c3f26196124794: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-cc935fc8b0984388a5db91f6e37824fa received from {username='slave'} at 127.5.70.7:53947 I1121 10:21:44.636682 10845 tablet_copy_service.cc:413] P cd693d36f0014dc3a7c3f26196124794: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-cc935fc8b0984388a5db91f6e37824fa on tablet cc935fc8b0984388a5db91f6e37824fa with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:44.637930 14980 tablet_copy_client.cc:420] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:44.638610 14980 tablet_bootstrap.cc:439] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:44.642055 14980 tablet_bootstrap.cc:439] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:44.642208 14980 tablet_bootstrap.cc:439] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:44.642300 14980 ts_tablet_manager.cc:1104] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I1121 10:21:44.642422 14980 raft_consensus.cc:337] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:44.642444 14980 raft_consensus.cc:702] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:44.642498 14980 consensus_queue.cc:231] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:44.642608 14980 ts_tablet_manager.cc:1132] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:21:44.642882 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-4dfbf2f060fb481fb74acc019b6eaebc received from {username='slave'} at 127.5.70.5:36527 I1121 10:21:44.642935 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-4dfbf2f060fb481fb74acc019b6eaebc on tablet 4dfbf2f060fb481fb74acc019b6eaebc with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:44.661025 15008 tablet_copy_client.cc:420] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:44.661949 15008 tablet_bootstrap.cc:439] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:44.665030 15008 tablet_bootstrap.cc:439] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:44.665244 15008 tablet_bootstrap.cc:439] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:44.665333 15008 ts_tablet_manager.cc:1104] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s I1121 10:21:44.665462 15008 raft_consensus.cc:337] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:44.665480 15008 raft_consensus.cc:702] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:44.665515 15008 consensus_queue.cc:231] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:44.665582 15008 ts_tablet_manager.cc:1132] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:44.665817 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-8bbbf8a8d2df4f76aa83b5fe842b29a6 received from {username='slave'} at 127.5.70.6:37621 I1121 10:21:44.665853 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-8bbbf8a8d2df4f76aa83b5fe842b29a6 on tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:44.707134 12389 raft_consensus.cc:1123] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:44.735630 15067 tablet_copy_client.cc:420] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:44.736380 15067 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:44.739766 15067 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:44.739928 15067 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:44.740001 15067 ts_tablet_manager.cc:1104] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s I1121 10:21:44.740125 15067 raft_consensus.cc:337] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:44.740144 15067 raft_consensus.cc:702] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:44.740180 15067 consensus_queue.cc:231] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:44.740285 15067 ts_tablet_manager.cc:1132] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.000s user 0.000s sys 0.003s I1121 10:21:44.740572 11218 tablet_copy_service.cc:325] P 5457a73f9ecb40ee9889513a8412937b: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-172c695aadee4de6ad0a5a6508b290f6 received from {username='slave'} at 127.5.70.6:39461 I1121 10:21:44.740605 11218 tablet_copy_service.cc:413] P 5457a73f9ecb40ee9889513a8412937b: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-172c695aadee4de6ad0a5a6508b290f6 on tablet 172c695aadee4de6ad0a5a6508b290f6 with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:44.760352 15068 tablet_copy_client.cc:420] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:44.761229 15068 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:44.763972 15068 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:44.764194 15068 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:44.764297 15068 ts_tablet_manager.cc:1104] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I1121 10:21:44.764442 15068 raft_consensus.cc:337] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:44.764466 15068 raft_consensus.cc:702] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:44.764559 15068 consensus_queue.cc:231] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:44.764811 15068 ts_tablet_manager.cc:1132] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:21:44.765053 11218 tablet_copy_service.cc:325] P 5457a73f9ecb40ee9889513a8412937b: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-df2330a22c8142ada6d056401d0a064d received from {username='slave'} at 127.5.70.8:35393 I1121 10:21:44.765080 11218 tablet_copy_service.cc:413] P 5457a73f9ecb40ee9889513a8412937b: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-df2330a22c8142ada6d056401d0a064d on tablet df2330a22c8142ada6d056401d0a064d with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:44.852274 12389 raft_consensus.cc:1123] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:44.854568 15069 tablet_copy_client.cc:420] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:44.855439 15069 tablet_bootstrap.cc:439] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:44.858599 15069 tablet_bootstrap.cc:439] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:44.858810 15069 tablet_bootstrap.cc:439] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:44.858894 15069 ts_tablet_manager.cc:1104] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I1121 10:21:44.859076 15069 raft_consensus.cc:337] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:44.859097 15069 raft_consensus.cc:702] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:44.859131 15069 consensus_queue.cc:231] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:44.859221 15069 ts_tablet_manager.cc:1132] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:44.859483 11218 tablet_copy_service.cc:325] P 5457a73f9ecb40ee9889513a8412937b: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-0e5ec5084c6a4d6089c2cb198367d342 received from {username='slave'} at 127.5.70.7:51745 I1121 10:21:44.859511 11218 tablet_copy_service.cc:413] P 5457a73f9ecb40ee9889513a8412937b: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-0e5ec5084c6a4d6089c2cb198367d342 on tablet 0e5ec5084c6a4d6089c2cb198367d342 with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:44.875680 12389 raft_consensus.cc:1123] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:45.008729 12513 raft_consensus.cc:1123] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:45.041126 12265 raft_consensus.cc:1123] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:45.081743 15120 tablet_copy_client.cc:420] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:45.082507 15120 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:45.085672 15120 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:45.085887 15120 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:45.085981 15120 ts_tablet_manager.cc:1104] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s I1121 10:21:45.086153 15120 raft_consensus.cc:337] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:45.086184 15120 raft_consensus.cc:702] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:45.086243 15120 consensus_queue.cc:231] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:45.086349 15120 ts_tablet_manager.cc:1132] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:45.086658 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-635b29a1f41e418faffe3677886048e5 received from {username='slave'} at 127.5.70.6:51069 I1121 10:21:45.086684 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-635b29a1f41e418faffe3677886048e5 on tablet 635b29a1f41e418faffe3677886048e5 with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:45.119566 15122 tablet_copy_client.cc:420] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:45.120436 15122 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:45.123718 15122 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:45.123945 15122 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:45.124043 15122 ts_tablet_manager.cc:1104] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s I1121 10:21:45.124198 15122 raft_consensus.cc:337] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:45.124219 15122 raft_consensus.cc:702] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:45.124274 15122 consensus_queue.cc:231] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:45.124358 15122 ts_tablet_manager.cc:1132] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:45.124611 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-69ac58ad89e447a494fc407037d27f8d received from {username='slave'} at 127.5.70.8:52727 I1121 10:21:45.124639 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-69ac58ad89e447a494fc407037d27f8d on tablet 69ac58ad89e447a494fc407037d27f8d with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:45.167748 12637 raft_consensus.cc:1123] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:45.178575 15173 tablet_copy_client.cc:420] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:45.179368 15173 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:45.182296 15173 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:45.182469 15173 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:45.182543 15173 ts_tablet_manager.cc:1104] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I1121 10:21:45.182655 15173 raft_consensus.cc:337] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:45.182675 15173 raft_consensus.cc:702] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:45.182723 15173 consensus_queue.cc:231] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:45.182801 15173 ts_tablet_manager.cc:1132] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:45.183120 11218 tablet_copy_service.cc:325] P 5457a73f9ecb40ee9889513a8412937b: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-5f7d9348eaaa4477b9dd581c28e2d5e3 received from {username='slave'} at 127.5.70.5:34817 I1121 10:21:45.183147 11218 tablet_copy_service.cc:413] P 5457a73f9ecb40ee9889513a8412937b: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-5f7d9348eaaa4477b9dd581c28e2d5e3 on tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:45.228309 12265 raft_consensus.cc:1123] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:45.289852 12389 raft_consensus.cc:1123] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:45.296689 12513 raft_consensus.cc:1123] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:45.307425 14372 raft_consensus.cc:971] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:45.307611 14372 consensus_queue.cc:208] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.308001 12389 raft_consensus.cc:1181] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.308043 10816 raft_consensus.cc:1181] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.308048 11202 raft_consensus.cc:1181] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.308243 14372 consensus_queue.cc:984] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.308331 15547 consensus_queue.cc:984] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.308425 15546 consensus_queue.cc:984] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.309507 15544 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.309669 10816 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.309747 11202 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.310750 12389 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.310776 10701 catalog_manager.cc:4143] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.312309 10953 consensus_queue.cc:208] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.312644 12389 raft_consensus.cc:1181] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.312697 10816 raft_consensus.cc:1181] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.312827 15546 consensus_queue.cc:984] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.313005 15546 consensus_queue.cc:984] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.313508 15544 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.313694 12389 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.313781 10816 raft_consensus.cc:2801] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.314532 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:45.314803 10709 catalog_manager.cc:4143] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.315661 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:45.315855 15555 tablet_replica.cc:282] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:45.315903 15555 raft_consensus.cc:2156] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:45.315917 15555 raft_consensus.cc:2183] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:45.316226 15555 ts_tablet_manager.cc:1457] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:45.317616 15555 ts_tablet_manager.cc:1470] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:45.317656 15555 log.cc:1153] T 8bbbf8a8d2df4f76aa83b5fe842b29a6 P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/8bbbf8a8d2df4f76aa83b5fe842b29a6 I1121 10:21:45.318127 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 (table rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]) successfully deleted I1121 10:21:45.327610 15557 raft_consensus.cc:971] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:45.327872 15557 consensus_queue.cc:208] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.328591 10816 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.328667 10953 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.328677 12389 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.328856 15556 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.329013 14016 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.329131 15559 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.329939 14016 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.330009 12637 raft_consensus.cc:1123] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:21:45.330230 10953 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.331396 12389 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.331439 10705 catalog_manager.cc:4143] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:45.331488 10816 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.332204 11202 consensus_queue.cc:208] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.332659 12389 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.332684 10816 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.332866 15556 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.333062 15556 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.333523 15559 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.333648 12389 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.333811 10816 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.334311 10693 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 172c695aadee4de6ad0a5a6508b290f6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:45.334619 10701 catalog_manager.cc:4143] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.335345 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 172c695aadee4de6ad0a5a6508b290f6 with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:45.335544 15564 tablet_replica.cc:282] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:45.335611 15564 raft_consensus.cc:2156] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:45.335638 15564 raft_consensus.cc:2183] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:45.335872 15564 ts_tablet_manager.cc:1457] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:45.337417 15564 ts_tablet_manager.cc:1470] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:45.337457 15564 log.cc:1153] T 172c695aadee4de6ad0a5a6508b290f6 P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/172c695aadee4de6ad0a5a6508b290f6 I1121 10:21:45.337852 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 172c695aadee4de6ad0a5a6508b290f6 (table rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]) successfully deleted I1121 10:21:45.383761 15556 raft_consensus.cc:971] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:45.383956 15556 consensus_queue.cc:208] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.384485 12389 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.384539 11080 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.384562 10816 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.384765 15556 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.384829 15557 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.384946 15559 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.386075 15556 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.386294 10816 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.386344 12389 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.387230 10709 catalog_manager.cc:4143] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:45.387532 11080 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.453945 15227 raft_consensus.cc:971] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:45.454190 15227 consensus_queue.cc:208] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:45.454716 12513 raft_consensus.cc:1181] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.454857 11202 raft_consensus.cc:1181] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.454867 11080 raft_consensus.cc:1181] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.454988 15569 consensus_queue.cc:984] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.455062 15227 consensus_queue.cc:984] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.455163 15572 consensus_queue.cc:984] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.456101 15569 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.456255 12513 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.456284 11205 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.457376 10705 catalog_manager.cc:4143] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce reported cstate change: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.458158 11080 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.458825 10816 consensus_queue.cc:208] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:45.459177 12513 raft_consensus.cc:1181] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.459357 11202 raft_consensus.cc:1181] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.459363 15571 consensus_queue.cc:984] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.459600 15576 consensus_queue.cc:984] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.459980 15578 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.460105 12513 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.460304 11202 raft_consensus.cc:2801] T cc935fc8b0984388a5db91f6e37824fa P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.460800 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet cc935fc8b0984388a5db91f6e37824fa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:45.461194 10709 catalog_manager.cc:4143] T cc935fc8b0984388a5db91f6e37824fa P 77830a7609a549e0a1c76586d0e301ce reported cstate change: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.461861 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet cc935fc8b0984388a5db91f6e37824fa with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:45.461987 15579 tablet_replica.cc:282] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:45.462038 15579 raft_consensus.cc:2156] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:45.462064 15579 raft_consensus.cc:2183] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:45.462302 15579 ts_tablet_manager.cc:1457] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:45.463573 15579 ts_tablet_manager.cc:1470] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:45.463614 15579 log.cc:1153] T cc935fc8b0984388a5db91f6e37824fa P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/cc935fc8b0984388a5db91f6e37824fa I1121 10:21:45.463951 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet cc935fc8b0984388a5db91f6e37824fa (table rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]) successfully deleted I1121 10:21:45.530577 11202 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" dest_uuid: "5457a73f9ecb40ee9889513a8412937b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:40800 I1121 10:21:45.530619 11202 raft_consensus.cc:581] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Received request to transfer leadership I1121 10:21:45.559391 11202 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" dest_uuid: "5457a73f9ecb40ee9889513a8412937b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:40818 I1121 10:21:45.559430 11202 raft_consensus.cc:581] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Received request to transfer leadership I1121 10:21:45.570096 15386 raft_consensus.cc:971] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:45.570338 15386 consensus_queue.cc:208] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:45.570803 11202 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.570817 10816 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.571004 12637 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.571053 15676 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.571127 15678 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.571344 15386 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.572245 15678 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.572394 10816 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.572638 12637 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.573746 10705 catalog_manager.cc:4143] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.574158 11202 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.574872 11080 consensus_queue.cc:208] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:45.575220 10816 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.575281 12637 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.575405 15386 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.575893 15386 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s W1121 10:21:45.576045 10690 catalog_manager.cc:3695] ChangeConfig:REMOVE_PEER RPC for tablet 69ac58ad89e447a494fc407037d27f8d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } }. Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:45.576058 15580 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.576272 10816 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.576725 12637 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.576956 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 69ac58ad89e447a494fc407037d27f8d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:45.577337 10709 catalog_manager.cc:4143] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.578012 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 69ac58ad89e447a494fc407037d27f8d with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:45.578116 15555 tablet_replica.cc:282] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:45.578173 15555 raft_consensus.cc:2156] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:45.578200 15555 raft_consensus.cc:2183] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:45.578425 15555 ts_tablet_manager.cc:1457] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:45.579668 15555 ts_tablet_manager.cc:1470] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:45.579702 15555 log.cc:1153] T 69ac58ad89e447a494fc407037d27f8d P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/69ac58ad89e447a494fc407037d27f8d I1121 10:21:45.580024 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 69ac58ad89e447a494fc407037d27f8d (table rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]) successfully deleted I1121 10:21:45.606335 15676 raft_consensus.cc:971] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:45.606648 15676 consensus_queue.cc:208] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:45.607218 11202 raft_consensus.cc:1181] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.607245 12265 raft_consensus.cc:1181] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.607331 10816 raft_consensus.cc:1181] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.607440 15386 consensus_queue.cc:984] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.607527 15676 consensus_queue.cc:984] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.607635 15678 consensus_queue.cc:984] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.608496 15676 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.608711 10816 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.608736 12265 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.609989 10705 catalog_manager.cc:4143] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.610276 11202 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.610888 11080 consensus_queue.cc:208] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:45.611495 11202 raft_consensus.cc:1181] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.611557 12265 raft_consensus.cc:1181] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.611642 15676 consensus_queue.cc:984] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.611737 15678 consensus_queue.cc:984] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.612327 15676 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.612471 12265 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.612592 11202 raft_consensus.cc:2801] T 4dfbf2f060fb481fb74acc019b6eaebc P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.613351 10701 catalog_manager.cc:4143] T 4dfbf2f060fb481fb74acc019b6eaebc P fed6a935d10042b48cdf1f30824f6c57 reported cstate change: config changed from index 5 to 6, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.613396 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 4dfbf2f060fb481fb74acc019b6eaebc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:45.614207 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet 4dfbf2f060fb481fb74acc019b6eaebc with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:45.614361 15686 tablet_replica.cc:282] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:45.614418 15686 raft_consensus.cc:2156] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:45.614442 15686 raft_consensus.cc:2183] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:45.614753 15686 ts_tablet_manager.cc:1457] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:45.616127 15686 ts_tablet_manager.cc:1470] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:45.616160 15686 log.cc:1153] T 4dfbf2f060fb481fb74acc019b6eaebc P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/4dfbf2f060fb481fb74acc019b6eaebc I1121 10:21:45.616547 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet 4dfbf2f060fb481fb74acc019b6eaebc (table rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]) successfully deleted I1121 10:21:45.640182 10690 catalog_manager.cc:3712] ChangeConfig:REMOVE_PEER RPC for tablet 69ac58ad89e447a494fc407037d27f8d with cas_config_opid_index 5: aborting the task: latest config opid_index 6; task opid_index 5 I1121 10:21:45.786541 11202 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" dest_uuid: "5457a73f9ecb40ee9889513a8412937b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:40834 I1121 10:21:45.786577 11202 raft_consensus.cc:581] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Received request to transfer leadership I1121 10:21:45.797605 15556 raft_consensus.cc:971] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:45.797817 15556 consensus_queue.cc:208] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:45.798306 12265 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.798317 10953 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.798326 11080 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.798512 15557 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.798604 15563 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.798787 15556 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.800487 15557 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.800735 12265 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.801901 10709 catalog_manager.cc:4143] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:45.801934 10953 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.802701 11070 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:45.802784 15563 raft_consensus.cc:900] Instructing follower ca8a596d4c4a4b438860e5e0420e3ad6 to start an election I1121 10:21:45.802826 15563 raft_consensus.cc:988] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Signalling peer ca8a596d4c4a4b438860e5e0420e3ad6to start an election I1121 10:21:45.803443 11080 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" from {username='slave'} at 127.5.70.4:41745 I1121 10:21:45.803479 11080 raft_consensus.cc:469] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:21:45.803491 11080 raft_consensus.cc:2905] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:45.804160 11080 raft_consensus.cc:491] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.804258 11080 leader_election.cc:296] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 2 election: Requested vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787) I1121 10:21:45.804709 10816 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:45.804747 10816 raft_consensus.cc:2905] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:45.804818 12389 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2c3332f15c014e68bed03f69d8c6287d" I1121 10:21:45.804857 12389 raft_consensus.cc:2905] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:45.805471 12389 raft_consensus.cc:2359] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 2. I1121 10:21:45.805493 10816 raft_consensus.cc:2359] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 2. I1121 10:21:45.805680 11011 leader_election.cc:310] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2c3332f15c014e68bed03f69d8c6287d, ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:45.805755 15678 raft_consensus.cc:2653] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:21:45.805785 15678 raft_consensus.cc:664] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 LEADER]: Becoming Leader. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:45.805866 15678 consensus_queue.cc:208] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.805922 11202 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "1fa5ac03774f4944815d262fcdffa2f6" candidate_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:45.805953 11202 raft_consensus.cc:2900] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:21:45.805980 11202 raft_consensus.cc:702] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Running, Role: LEADER I1121 10:21:45.806049 11202 consensus_queue.cc:231] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.806097 11202 raft_consensus.cc:2905] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:45.806573 11202 raft_consensus.cc:2359] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca8a596d4c4a4b438860e5e0420e3ad6 in term 2. I1121 10:21:45.807018 10705 catalog_manager.cc:4143] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: term changed from 1 to 2, leader changed from 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) to ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3). New cstate: current_term: 2 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:45.807554 15563 raft_consensus.cc:971] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:45.807706 15563 consensus_queue.cc:208] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:45.808125 12637 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.808152 10816 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.808321 14016 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.808404 14016 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.808599 10953 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.808859 15557 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.809749 14016 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.809895 12637 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.810741 10953 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.810746 10709 catalog_manager.cc:4143] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:45.810840 10816 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.811384 11205 consensus_queue.cc:208] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:45.811725 12637 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.811841 10816 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.811916 15556 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.811993 15556 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.812556 15559 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.812708 12637 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.812745 10816 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:45.813596 10705 catalog_manager.cc:4143] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 5 to 6, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:45.813735 10693 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet df2330a22c8142ada6d056401d0a064d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:45.814206 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet df2330a22c8142ada6d056401d0a064d with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:45.814265 15564 tablet_replica.cc:282] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:45.814312 15564 raft_consensus.cc:2156] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:45.814334 15564 raft_consensus.cc:2183] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:45.814726 15564 ts_tablet_manager.cc:1457] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:45.816239 15564 ts_tablet_manager.cc:1470] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:45.816277 15564 log.cc:1153] T df2330a22c8142ada6d056401d0a064d P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/df2330a22c8142ada6d056401d0a064d I1121 10:21:45.816625 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet df2330a22c8142ada6d056401d0a064d (table rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]) successfully deleted I1121 10:21:45.824580 11205 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" dest_uuid: "5457a73f9ecb40ee9889513a8412937b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:40852 I1121 10:21:45.824616 11205 raft_consensus.cc:581] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Received request to transfer leadership I1121 10:21:45.836577 15563 raft_consensus.cc:971] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:45.836791 15563 consensus_queue.cc:208] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:45.837177 15580 raft_consensus.cc:971] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:45.837249 11080 raft_consensus.cc:1181] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.837337 15580 consensus_queue.cc:208] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.837332 10816 raft_consensus.cc:1181] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.837410 12513 raft_consensus.cc:1181] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.837585 15556 consensus_queue.cc:984] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.837669 14016 consensus_queue.cc:984] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.837759 10953 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.837771 15563 consensus_queue.cc:984] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.837834 12389 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.837846 11205 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:21:45.838003 15386 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.838119 15678 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.838201 15676 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:21:45.839017 15556 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.839363 15678 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.839663 10816 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.840339 12513 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.840374 10709 catalog_manager.cc:4143] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:45.840781 12389 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.841758 11205 consensus_queue.cc:208] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:45.841768 10701 catalog_manager.cc:4143] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:45.842166 10816 raft_consensus.cc:1181] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.842430 10953 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.842486 15563 consensus_queue.cc:984] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.842981 12513 raft_consensus.cc:1181] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.843365 11080 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.843379 15563 consensus_queue.cc:984] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.844823 11205 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.844292 14016 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.845875 10693 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 0e5ec5084c6a4d6089c2cb198367d342 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:45.845960 12513 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.846247 10816 raft_consensus.cc:2801] T 0e5ec5084c6a4d6089c2cb198367d342 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:45.846472 11080 consensus_queue.cc:208] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:45.847023 10953 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.847074 10709 catalog_manager.cc:4143] T 0e5ec5084c6a4d6089c2cb198367d342 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 5 to 6, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:45.847515 15580 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.847532 12389 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:21:45.847692 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 0e5ec5084c6a4d6089c2cb198367d342 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:45.847755 15579 tablet_replica.cc:282] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:45.847774 15676 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:45.847800 15579 raft_consensus.cc:2156] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:45.847817 15579 raft_consensus.cc:2183] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:45.848114 15580 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.848302 15579 ts_tablet_manager.cc:1457] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:45.848621 12389 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.848810 10953 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.849862 10701 catalog_manager.cc:4143] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 5 to 6, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:45.850024 15579 ts_tablet_manager.cc:1470] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:45.850067 15579 log.cc:1153] T 0e5ec5084c6a4d6089c2cb198367d342 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/0e5ec5084c6a4d6089c2cb198367d342 I1121 10:21:45.850149 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 635b29a1f41e418faffe3677886048e5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:45.850437 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 635b29a1f41e418faffe3677886048e5 with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:45.850504 15555 tablet_replica.cc:282] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:45.850526 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 0e5ec5084c6a4d6089c2cb198367d342 (table rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]) successfully deleted I1121 10:21:45.850550 15555 raft_consensus.cc:2156] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:45.850569 15555 raft_consensus.cc:2183] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:45.850875 15555 ts_tablet_manager.cc:1457] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:45.852182 15555 ts_tablet_manager.cc:1470] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:45.852216 15555 log.cc:1153] T 635b29a1f41e418faffe3677886048e5 P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/635b29a1f41e418faffe3677886048e5 I1121 10:21:45.852576 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 635b29a1f41e418faffe3677886048e5 (table rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]) successfully deleted I1121 10:21:46.040347 11205 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" dest_uuid: "5457a73f9ecb40ee9889513a8412937b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:40862 I1121 10:21:46.040393 11205 raft_consensus.cc:581] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Received request to transfer leadership I1121 10:21:46.056970 11205 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" dest_uuid: "5457a73f9ecb40ee9889513a8412937b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:40872 I1121 10:21:46.057026 11205 raft_consensus.cc:581] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Received request to transfer leadership I1121 10:21:46.203312 10816 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I1121 10:21:46.203580 15676 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:46.267086 11205 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" dest_uuid: "5457a73f9ecb40ee9889513a8412937b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:40878 I1121 10:21:46.267127 11205 raft_consensus.cc:581] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Received request to transfer leadership I1121 10:21:46.300793 12389 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I1121 10:21:46.301154 15676 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:46.302533 11205 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I1121 10:21:46.302906 15580 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:46.303112 11080 consensus_queue.cc:208] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:46.303478 10816 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch) I1121 10:21:46.303668 15386 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.303886 12389 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch) I1121 10:21:46.304153 15386 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.304358 15580 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:46.304958 12389 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:46.305508 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 1fa5ac03774f4944815d262fcdffa2f6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:46.305625 10705 catalog_manager.cc:4143] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 5 to 7, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 2 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:46.306107 10816 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:46.306483 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 1fa5ac03774f4944815d262fcdffa2f6 with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:46.306552 15555 tablet_replica.cc:282] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:46.306602 15555 raft_consensus.cc:2156] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:46.306615 15555 raft_consensus.cc:2183] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:46.306876 15555 ts_tablet_manager.cc:1457] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:46.308410 15555 ts_tablet_manager.cc:1470] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I1121 10:21:46.308444 15555 log.cc:1153] T 1fa5ac03774f4944815d262fcdffa2f6 P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/1fa5ac03774f4944815d262fcdffa2f6 I1121 10:21:46.308858 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 1fa5ac03774f4944815d262fcdffa2f6 (table rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]) successfully deleted I1121 10:21:46.325955 15563 raft_consensus.cc:900] Instructing follower af0d02c642ed4eab8d971d9f05d32044 to start an election I1121 10:21:46.326004 15563 raft_consensus.cc:988] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Signalling peer af0d02c642ed4eab8d971d9f05d32044to start an election I1121 10:21:46.326210 10953 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" from {username='slave'} at 127.5.70.4:35981 I1121 10:21:46.326242 10953 raft_consensus.cc:469] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:21:46.326251 10953 raft_consensus.cc:2905] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:46.326942 10953 raft_consensus.cc:491] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:46.327041 10953 leader_election.cc:296] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 2 election: Requested vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757) I1121 10:21:46.327239 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:46.327273 11205 raft_consensus.cc:2900] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:21:46.327283 11205 raft_consensus.cc:702] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5457a73f9ecb40ee9889513a8412937b, State: Running, Role: LEADER I1121 10:21:46.327273 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:46.327317 11080 raft_consensus.cc:2905] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:46.327360 12265 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5f7d9348eaaa4477b9dd581c28e2d5e3" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fed6a935d10042b48cdf1f30824f6c57" I1121 10:21:46.327368 11205 consensus_queue.cc:231] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:46.327414 11205 raft_consensus.cc:2905] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:46.327404 12265 raft_consensus.cc:2905] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:46.328070 11205 raft_consensus.cc:2359] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 2. I1121 10:21:46.328128 11080 raft_consensus.cc:2359] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 2. I1121 10:21:46.328128 12265 raft_consensus.cc:2359] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 2. I1121 10:21:46.328301 10887 leader_election.cc:310] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:46.328379 15546 raft_consensus.cc:2653] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:21:46.328460 15546 raft_consensus.cc:664] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 2 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:46.328594 15546 consensus_queue.cc:208] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:46.329277 10705 catalog_manager.cc:4143] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 1 to 2, leader changed from 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 2 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:46.761847 12265 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I1121 10:21:46.762115 15852 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:46.774796 11080 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I1121 10:21:46.775059 15852 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:21:46.776479 11205 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I1121 10:21:46.776978 10953 consensus_queue.cc:208] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:46.777420 12265 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch) I1121 10:21:46.777417 11080 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch) I1121 10:21:46.777623 15876 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.777686 15852 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.778724 15546 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:46.778934 12265 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:46.778962 11080 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:46.779685 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:46.779973 10709 catalog_manager.cc:4143] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 5 to 7, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 2 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:46.780975 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:46.781091 15555 tablet_replica.cc:282] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:46.781148 15555 raft_consensus.cc:2156] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:46.781172 15555 raft_consensus.cc:2183] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:46.781502 15555 ts_tablet_manager.cc:1457] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:46.782981 15555 ts_tablet_manager.cc:1470] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:21:46.783023 15555 log.cc:1153] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/5f7d9348eaaa4477b9dd581c28e2d5e3 I1121 10:21:46.783457 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 (table rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]) successfully deleted I1121 10:21:46.934993 11205 consensus_queue.cc:208] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:46.935541 10816 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.935778 15936 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.935828 12389 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.936038 15936 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.936657 15563 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:46.937032 12389 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:46.937924 10816 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:46.938112 10701 catalog_manager.cc:4143] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W1121 10:21:46.939182 11136 consensus_peers.cc:473] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 172c695aadee4de6ad0a5a6508b290f6. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:46.942885 11080 consensus_queue.cc:208] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:46.943452 12513 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.943512 11205 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:46.943614 11012 consensus_peers.cc:473] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b34e50304ff46e59622de2e979ac7b8. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:46.943769 15948 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.943883 15947 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.944604 15676 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:46.944777 11205 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:46.944806 12513 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:46.945770 10705 catalog_manager.cc:4143] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 6 to 7, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:46.949512 11080 consensus_queue.cc:208] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:46.950053 10953 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.950075 12389 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:46.950253 11011 consensus_peers.cc:473] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 635b29a1f41e418faffe3677886048e5. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:46.950284 15947 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.950351 15951 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.951153 15951 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:46.951292 10953 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:46.951324 12389 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:46.952406 10701 catalog_manager.cc:4143] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:46.956068 10953 consensus_queue.cc:208] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:46.956465 11080 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch) I1121 10:21:46.956503 12265 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch) W1121 10:21:46.956609 10888 consensus_peers.cc:473] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 -> Peer 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787): Couldn't send request to peer 2c3332f15c014e68bed03f69d8c6287d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f7d9348eaaa4477b9dd581c28e2d5e3. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:46.956766 15546 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:46.956840 15852 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:46.957429 15546 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:46.957603 11080 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:46.957804 12265 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } } I1121 10:21:46.958580 10705 catalog_manager.cc:4143] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 7 to 8, NON_VOTER 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) added. New cstate: current_term: 2 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:46.962975 10816 consensus_queue.cc:208] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:46.963603 11205 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.963614 12265 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:46.963641 10763 consensus_peers.cc:473] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7de6e746a8246f89d8b65b59803c3af. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:46.963810 15973 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.963884 15576 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.964674 15576 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:46.964843 11205 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:46.964881 12265 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:46.965925 10701 catalog_manager.cc:4143] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 reported cstate change: config changed from index 6 to 7, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:46.970312 11080 consensus_queue.cc:208] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:46.970904 10816 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.971038 12513 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.971303 15947 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.971422 15951 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.972121 15947 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:46.972254 10816 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:46.973515 10705 catalog_manager.cc:4143] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:46.973573 12513 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } W1121 10:21:46.975539 11012 consensus_peers.cc:473] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a40d6cf45df24788adc85f238c9bcc7e. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:46.978307 10953 consensus_queue.cc:208] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:46.978832 12265 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.978839 10816 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:46.978983 10888 consensus_peers.cc:473] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25b55ff241d8442897ea97523a2cd6fd. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:46.979061 15852 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.979154 15876 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.980015 15546 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:46.980144 10816 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:46.980746 12265 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:46.981070 10705 catalog_manager.cc:4143] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 6 to 7, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:46.985149 11080 consensus_queue.cc:208] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:46.985674 12637 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:46.985747 11011 consensus_peers.cc:473] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69ac58ad89e447a494fc407037d27f8d. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:46.985700 10816 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.985990 15948 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.986078 15947 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.986846 15948 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:46.986953 10816 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:46.987001 12637 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:46.987977 10702 catalog_manager.cc:4143] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:46.992427 10953 consensus_queue.cc:208] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:46.993016 11205 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.993193 12389 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:46.993345 15993 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.993448 15876 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:46.994844 15876 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:46.995136 12389 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } W1121 10:21:46.995779 10888 consensus_peers.cc:473] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69c25581de6e4945a167245a4dcbbfe0. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:46.996050 10702 catalog_manager.cc:4143] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:46.996165 11205 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:47.000128 10953 consensus_queue.cc:208] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:47.000682 12389 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:47.000871 15852 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:47.000872 10816 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:47.000938 10888 consensus_peers.cc:473] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 114dbe156d634fca957db3164f8afabb. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:47.001055 15852 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:47.001760 15546 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:47.001924 12389 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:47.001947 10816 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:47.002715 10705 catalog_manager.cc:4143] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 6 to 7, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:47.007220 11080 consensus_queue.cc:208] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:47.007705 10953 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:47.007783 12389 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:47.007854 11012 consensus_peers.cc:473] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6765d5e95b8f45d4a8f0bb9e5d76b9f0. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:47.007956 15947 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:47.008057 15947 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:47.008776 15948 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:47.008916 10953 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:47.009039 12389 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:47.010119 10701 catalog_manager.cc:4143] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:47.014348 10816 consensus_queue.cc:208] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:47.014999 12637 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:47.015069 10953 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:47.015117 10762 consensus_peers.cc:473] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 -> Peer fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757): Couldn't send request to peer fed6a935d10042b48cdf1f30824f6c57. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e0bc1c59a1b4e2c9e761a6de8085226. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:47.015260 15576 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:47.015365 15973 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:47.016121 15576 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:47.016255 12637 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:47.016400 10953 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:47.017518 10705 catalog_manager.cc:4143] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 6 to 7, NON_VOTER fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) added. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } I1121 10:21:47.021618 11080 consensus_queue.cc:208] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:47.022014 12389 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch) I1121 10:21:47.022109 10816 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch) W1121 10:21:47.022192 11012 consensus_peers.cc:473] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471): Couldn't send request to peer 97a8ceca326a4f45b15a5b9c4b4857fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fa5ac03774f4944815d262fcdffa2f6. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:47.022338 15948 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:47.022408 15947 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:47.023041 15951 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:47.023288 10816 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:47.023285 12389 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } } I1121 10:21:47.024358 10702 catalog_manager.cc:4143] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 7 to 8, NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) added. New cstate: current_term: 2 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:47.028218 11080 consensus_queue.cc:208] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:47.028687 12265 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:47.028694 10953 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:47.028893 15947 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:47.028957 15948 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s W1121 10:21:47.029054 11012 consensus_peers.cc:473] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa85115a8bbd45dbb7ce1f18b923022a. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:47.029743 15947 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:47.029928 10953 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:47.029990 12265 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:47.030946 10701 catalog_manager.cc:4143] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:47.401597 16189 ts_tablet_manager.cc:721] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:47.401660 16189 tablet_copy_client.cc:243] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:47.404250 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet a40d6cf45df24788adc85f238c9bcc7e from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:41607) I1121 10:21:47.404287 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet a40d6cf45df24788adc85f238c9bcc7e from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:41607: session id = 80d2057cd4814fd3bfe268270d1abdd4-a40d6cf45df24788adc85f238c9bcc7e I1121 10:21:47.404804 11085 tablet_copy_source_session.cc:204] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.405586 16189 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet a40d6cf45df24788adc85f238c9bcc7e. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.407685 16189 tablet_copy_client.cc:581] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.407788 16189 tablet_copy_client.cc:544] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.408349 16189 tablet_copy_client.cc:755] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.412678 16191 ts_tablet_manager.cc:721] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:47.412784 16191 tablet_copy_client.cc:243] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:47.414503 16192 ts_tablet_manager.cc:721] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:47.414541 16192 tablet_copy_client.cc:243] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:43035 I1121 10:21:47.414827 10845 tablet_copy_service.cc:135] P cd693d36f0014dc3a7c3f26196124794: Received BeginTabletCopySession request for tablet c7de6e746a8246f89d8b65b59803c3af from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:53947) I1121 10:21:47.414857 10845 tablet_copy_service.cc:156] P cd693d36f0014dc3a7c3f26196124794: Beginning new tablet copy session on tablet c7de6e746a8246f89d8b65b59803c3af from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:53947: session id = 77830a7609a549e0a1c76586d0e301ce-c7de6e746a8246f89d8b65b59803c3af I1121 10:21:47.415246 10845 tablet_copy_source_session.cc:204] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.415670 16192 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet c7de6e746a8246f89d8b65b59803c3af. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.415726 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 69c25581de6e4945a167245a4dcbbfe0 from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:35497) I1121 10:21:47.415769 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 69c25581de6e4945a167245a4dcbbfe0 from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:35497: session id = 80d2057cd4814fd3bfe268270d1abdd4-69c25581de6e4945a167245a4dcbbfe0 I1121 10:21:47.416175 10977 tablet_copy_source_session.cc:204] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.416499 16191 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69c25581de6e4945a167245a4dcbbfe0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.416682 16192 tablet_copy_client.cc:581] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.416786 16192 tablet_copy_client.cc:544] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.417346 16192 tablet_copy_client.cc:755] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.417585 16191 tablet_copy_client.cc:581] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.417690 16191 tablet_copy_client.cc:544] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.418162 16191 tablet_copy_client.cc:755] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.430701 16193 ts_tablet_manager.cc:721] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:47.430748 16193 tablet_copy_client.cc:243] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:47.430936 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 25b55ff241d8442897ea97523a2cd6fd from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:59669) I1121 10:21:47.430976 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 25b55ff241d8442897ea97523a2cd6fd from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:59669: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-25b55ff241d8442897ea97523a2cd6fd I1121 10:21:47.431337 10977 tablet_copy_source_session.cc:204] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.431694 16193 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 25b55ff241d8442897ea97523a2cd6fd. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.432638 16193 tablet_copy_client.cc:581] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.432746 16193 tablet_copy_client.cc:544] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.433207 16193 tablet_copy_client.cc:755] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.442309 16194 ts_tablet_manager.cc:721] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:47.442394 16194 tablet_copy_client.cc:243] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:37207 I1121 10:21:47.444921 11218 tablet_copy_service.cc:135] P 5457a73f9ecb40ee9889513a8412937b: Received BeginTabletCopySession request for tablet 172c695aadee4de6ad0a5a6508b290f6 from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:44295) I1121 10:21:47.444994 11218 tablet_copy_service.cc:156] P 5457a73f9ecb40ee9889513a8412937b: Beginning new tablet copy session on tablet 172c695aadee4de6ad0a5a6508b290f6 from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:44295: session id = 80d2057cd4814fd3bfe268270d1abdd4-172c695aadee4de6ad0a5a6508b290f6 I1121 10:21:47.445541 11218 tablet_copy_source_session.cc:204] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.445982 16194 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 172c695aadee4de6ad0a5a6508b290f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.446866 16194 tablet_copy_client.cc:581] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.446964 16194 tablet_copy_client.cc:544] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.447566 16194 tablet_copy_client.cc:755] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.455562 16195 ts_tablet_manager.cc:721] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:47.455615 16195 tablet_copy_client.cc:243] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:47.455870 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 1fa5ac03774f4944815d262fcdffa2f6 from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:52727) I1121 10:21:47.455904 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 1fa5ac03774f4944815d262fcdffa2f6 from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:52727: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-1fa5ac03774f4944815d262fcdffa2f6 I1121 10:21:47.456553 11085 tablet_copy_source_session.cc:204] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.456914 16195 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 1fa5ac03774f4944815d262fcdffa2f6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.457907 16195 tablet_copy_client.cc:581] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.457998 16195 tablet_copy_client.cc:544] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.458528 16195 tablet_copy_client.cc:755] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.517858 16196 ts_tablet_manager.cc:721] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:47.517925 16196 tablet_copy_client.cc:243] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:47.518126 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 69ac58ad89e447a494fc407037d27f8d from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:36527) I1121 10:21:47.518157 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 69ac58ad89e447a494fc407037d27f8d from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:36527: session id = fed6a935d10042b48cdf1f30824f6c57-69ac58ad89e447a494fc407037d27f8d I1121 10:21:47.518673 11085 tablet_copy_source_session.cc:204] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.519071 16196 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 69ac58ad89e447a494fc407037d27f8d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.520153 16196 tablet_copy_client.cc:581] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.520236 16197 ts_tablet_manager.cc:721] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:47.520287 16196 tablet_copy_client.cc:544] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.520336 16197 tablet_copy_client.cc:243] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:47.520582 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet fa85115a8bbd45dbb7ce1f18b923022a from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:41607) I1121 10:21:47.520612 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet fa85115a8bbd45dbb7ce1f18b923022a from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:41607: session id = 80d2057cd4814fd3bfe268270d1abdd4-fa85115a8bbd45dbb7ce1f18b923022a I1121 10:21:47.520920 16196 tablet_copy_client.cc:755] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.520968 11085 tablet_copy_source_session.cc:204] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.521495 16197 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet fa85115a8bbd45dbb7ce1f18b923022a. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.522368 16197 tablet_copy_client.cc:581] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.522469 16197 tablet_copy_client.cc:544] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.523034 16197 tablet_copy_client.cc:755] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.550480 16243 ts_tablet_manager.cc:721] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:47.550534 16243 tablet_copy_client.cc:243] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:47.550734 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 635b29a1f41e418faffe3677886048e5 from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:36527) I1121 10:21:47.550768 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 635b29a1f41e418faffe3677886048e5 from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:36527: session id = fed6a935d10042b48cdf1f30824f6c57-635b29a1f41e418faffe3677886048e5 I1121 10:21:47.551354 11085 tablet_copy_source_session.cc:204] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.551754 16243 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 635b29a1f41e418faffe3677886048e5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.553232 16243 tablet_copy_client.cc:581] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.553335 16243 tablet_copy_client.cc:544] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.553862 16243 tablet_copy_client.cc:755] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.561568 16259 ts_tablet_manager.cc:721] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:47.561616 16259 tablet_copy_client.cc:243] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:47.561830 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 from peer 2c3332f15c014e68bed03f69d8c6287d ({username='slave'} at 127.5.70.6:37621) I1121 10:21:47.561864 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 from peer 2c3332f15c014e68bed03f69d8c6287d at {username='slave'} at 127.5.70.6:37621: session id = 2c3332f15c014e68bed03f69d8c6287d-5f7d9348eaaa4477b9dd581c28e2d5e3 I1121 10:21:47.562258 10977 tablet_copy_source_session.cc:204] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.562629 16259 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.563943 16259 tablet_copy_client.cc:581] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.564038 16259 tablet_copy_client.cc:544] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.564615 16259 tablet_copy_client.cc:755] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.566103 16269 ts_tablet_manager.cc:721] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:47.566157 16269 tablet_copy_client.cc:243] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:47.566335 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 7b34e50304ff46e59622de2e979ac7b8 from peer 97a8ceca326a4f45b15a5b9c4b4857fd ({username='slave'} at 127.5.70.8:52727) I1121 10:21:47.566371 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 7b34e50304ff46e59622de2e979ac7b8 from peer 97a8ceca326a4f45b15a5b9c4b4857fd at {username='slave'} at 127.5.70.8:52727: session id = 97a8ceca326a4f45b15a5b9c4b4857fd-7b34e50304ff46e59622de2e979ac7b8 I1121 10:21:47.566836 11085 tablet_copy_source_session.cc:204] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.567162 16269 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 7b34e50304ff46e59622de2e979ac7b8. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.568362 16269 tablet_copy_client.cc:581] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.568548 16269 tablet_copy_client.cc:544] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.569084 16269 tablet_copy_client.cc:755] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.576781 16271 ts_tablet_manager.cc:721] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: Initiating tablet copy from peer cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:47.576820 16271 tablet_copy_client.cc:243] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:43035 I1121 10:21:47.577040 10845 tablet_copy_service.cc:135] P cd693d36f0014dc3a7c3f26196124794: Received BeginTabletCopySession request for tablet 3e0bc1c59a1b4e2c9e761a6de8085226 from peer fed6a935d10042b48cdf1f30824f6c57 ({username='slave'} at 127.5.70.5:38887) I1121 10:21:47.577072 10845 tablet_copy_service.cc:156] P cd693d36f0014dc3a7c3f26196124794: Beginning new tablet copy session on tablet 3e0bc1c59a1b4e2c9e761a6de8085226 from peer fed6a935d10042b48cdf1f30824f6c57 at {username='slave'} at 127.5.70.5:38887: session id = fed6a935d10042b48cdf1f30824f6c57-3e0bc1c59a1b4e2c9e761a6de8085226 I1121 10:21:47.577512 10845 tablet_copy_source_session.cc:204] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.577913 16271 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 3e0bc1c59a1b4e2c9e761a6de8085226. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.578779 16271 tablet_copy_client.cc:581] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.578857 16271 tablet_copy_client.cc:544] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.579329 16271 tablet_copy_client.cc:755] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.604032 16272 ts_tablet_manager.cc:721] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:47.604084 16272 tablet_copy_client.cc:243] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:47.604336 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:41607) I1121 10:21:47.604370 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:41607: session id = 80d2057cd4814fd3bfe268270d1abdd4-6765d5e95b8f45d4a8f0bb9e5d76b9f0 I1121 10:21:47.604909 11085 tablet_copy_source_session.cc:204] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.605311 16272 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.606148 16272 tablet_copy_client.cc:581] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.606251 16272 tablet_copy_client.cc:544] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.606817 16272 tablet_copy_client.cc:755] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:47.620999 16273 ts_tablet_manager.cc:721] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:47.621043 16273 tablet_copy_client.cc:243] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:47.621313 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 114dbe156d634fca957db3164f8afabb from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:43411) I1121 10:21:47.621354 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 114dbe156d634fca957db3164f8afabb from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:43411: session id = 77830a7609a549e0a1c76586d0e301ce-114dbe156d634fca957db3164f8afabb I1121 10:21:47.621850 10977 tablet_copy_source_session.cc:204] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:47.622270 16273 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 114dbe156d634fca957db3164f8afabb. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:47.623276 16273 tablet_copy_client.cc:581] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:47.623379 16273 tablet_copy_client.cc:544] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:47.624038 16273 tablet_copy_client.cc:755] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:48.908576 16189 tablet_copy_client.cc:420] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:48.909445 16189 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:48.910732 16189 log.cc:810] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: Log is configured to *not* fsync() on all Append() calls I1121 10:21:48.914005 16189 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:48.914218 16189 tablet_bootstrap.cc:439] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:48.914324 16189 ts_tablet_manager.cc:1104] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s I1121 10:21:48.915114 16189 raft_consensus.cc:337] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:48.915163 16189 raft_consensus.cc:702] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:48.915262 16189 consensus_queue.cc:231] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:48.915457 16189 ts_tablet_manager.cc:1132] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s I1121 10:21:48.915844 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-a40d6cf45df24788adc85f238c9bcc7e received from {username='slave'} at 127.5.70.9:41607 I1121 10:21:48.915880 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-a40d6cf45df24788adc85f238c9bcc7e on tablet a40d6cf45df24788adc85f238c9bcc7e with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:48.917471 16192 tablet_copy_client.cc:420] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:48.918123 16192 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:48.918282 16191 tablet_copy_client.cc:420] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:48.918841 16191 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:48.921591 16192 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:48.921779 16192 tablet_bootstrap.cc:439] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:48.921865 16192 ts_tablet_manager.cc:1104] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I1121 10:21:48.921996 16192 raft_consensus.cc:337] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:48.922017 16192 raft_consensus.cc:702] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:48.922070 16192 consensus_queue.cc:231] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:48.922211 16192 ts_tablet_manager.cc:1132] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:21:48.922418 10845 tablet_copy_service.cc:325] P cd693d36f0014dc3a7c3f26196124794: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-c7de6e746a8246f89d8b65b59803c3af received from {username='slave'} at 127.5.70.7:53947 I1121 10:21:48.922451 10845 tablet_copy_service.cc:413] P cd693d36f0014dc3a7c3f26196124794: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-c7de6e746a8246f89d8b65b59803c3af on tablet c7de6e746a8246f89d8b65b59803c3af with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:48.922925 16191 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:48.923065 16191 tablet_bootstrap.cc:439] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:48.923149 16191 ts_tablet_manager.cc:1104] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s I1121 10:21:48.923300 16191 raft_consensus.cc:337] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:48.923319 16191 raft_consensus.cc:702] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:48.923352 16191 consensus_queue.cc:231] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:48.923426 16191 ts_tablet_manager.cc:1132] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:48.923632 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-69c25581de6e4945a167245a4dcbbfe0 received from {username='slave'} at 127.5.70.9:35497 I1121 10:21:48.923657 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-69c25581de6e4945a167245a4dcbbfe0 on tablet 69c25581de6e4945a167245a4dcbbfe0 with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:48.933349 16193 tablet_copy_client.cc:420] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:48.934020 16193 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:48.937439 16193 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:48.937620 16193 tablet_bootstrap.cc:439] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:48.937710 16193 ts_tablet_manager.cc:1104] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s I1121 10:21:48.937855 16193 raft_consensus.cc:337] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:48.937875 16193 raft_consensus.cc:702] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:48.937923 16193 consensus_queue.cc:231] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:48.938035 16193 ts_tablet_manager.cc:1132] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:21:48.938283 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-25b55ff241d8442897ea97523a2cd6fd received from {username='slave'} at 127.5.70.8:59669 I1121 10:21:48.938313 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-25b55ff241d8442897ea97523a2cd6fd on tablet 25b55ff241d8442897ea97523a2cd6fd with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:48.947715 16194 tablet_copy_client.cc:420] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:48.948294 16194 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:48.951872 16194 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:48.952041 16194 tablet_bootstrap.cc:439] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:48.952127 16194 ts_tablet_manager.cc:1104] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s I1121 10:21:48.952262 16194 raft_consensus.cc:337] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:48.952284 16194 raft_consensus.cc:702] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:48.952337 16194 consensus_queue.cc:231] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:48.952411 16194 ts_tablet_manager.cc:1132] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:48.952654 11218 tablet_copy_service.cc:325] P 5457a73f9ecb40ee9889513a8412937b: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-172c695aadee4de6ad0a5a6508b290f6 received from {username='slave'} at 127.5.70.9:44295 I1121 10:21:48.952687 11218 tablet_copy_service.cc:413] P 5457a73f9ecb40ee9889513a8412937b: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-172c695aadee4de6ad0a5a6508b290f6 on tablet 172c695aadee4de6ad0a5a6508b290f6 with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:48.958647 16195 tablet_copy_client.cc:420] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:48.959265 16195 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:48.962680 16195 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:48.962884 16195 tablet_bootstrap.cc:439] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:48.962955 16195 ts_tablet_manager.cc:1104] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s I1121 10:21:48.963071 16195 raft_consensus.cc:337] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:48.963093 16195 raft_consensus.cc:702] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:48.963136 16195 consensus_queue.cc:231] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:48.963279 16195 ts_tablet_manager.cc:1132] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:48.963560 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-1fa5ac03774f4944815d262fcdffa2f6 received from {username='slave'} at 127.5.70.8:52727 I1121 10:21:48.963591 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-1fa5ac03774f4944815d262fcdffa2f6 on tablet 1fa5ac03774f4944815d262fcdffa2f6 with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:49.021062 16196 tablet_copy_client.cc:420] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:49.022127 16196 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:49.023277 16197 tablet_copy_client.cc:420] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:49.024019 16197 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:49.029536 16197 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:49.029788 16197 tablet_bootstrap.cc:439] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:49.029891 16197 ts_tablet_manager.cc:1104] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s I1121 10:21:49.030071 16197 raft_consensus.cc:337] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:49.030100 16197 raft_consensus.cc:702] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:49.030159 16197 consensus_queue.cc:231] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:49.030236 16196 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:49.030345 16197 ts_tablet_manager.cc:1132] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:21:49.030506 16196 tablet_bootstrap.cc:439] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:49.030522 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-fa85115a8bbd45dbb7ce1f18b923022a received from {username='slave'} at 127.5.70.9:41607 I1121 10:21:49.030542 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-fa85115a8bbd45dbb7ce1f18b923022a on tablet fa85115a8bbd45dbb7ce1f18b923022a with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:49.030606 16196 ts_tablet_manager.cc:1104] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s I1121 10:21:49.030779 16196 raft_consensus.cc:337] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:49.030812 16196 raft_consensus.cc:702] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:49.030872 16196 consensus_queue.cc:231] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:49.030997 16196 ts_tablet_manager.cc:1132] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:21:49.031232 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-69ac58ad89e447a494fc407037d27f8d received from {username='slave'} at 127.5.70.5:36527 I1121 10:21:49.031260 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-69ac58ad89e447a494fc407037d27f8d on tablet 69ac58ad89e447a494fc407037d27f8d with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:49.054024 16243 tablet_copy_client.cc:420] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:49.054783 16243 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:49.058550 16243 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:49.058722 16243 tablet_bootstrap.cc:439] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:49.058818 16243 ts_tablet_manager.cc:1104] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s I1121 10:21:49.058948 16243 raft_consensus.cc:337] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:49.058970 16243 raft_consensus.cc:702] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:49.059029 16243 consensus_queue.cc:231] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:49.059124 16243 ts_tablet_manager.cc:1132] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:49.059321 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-635b29a1f41e418faffe3677886048e5 received from {username='slave'} at 127.5.70.5:36527 I1121 10:21:49.059355 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-635b29a1f41e418faffe3677886048e5 on tablet 635b29a1f41e418faffe3677886048e5 with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:49.064779 16259 tablet_copy_client.cc:420] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:49.065528 16259 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap starting. I1121 10:21:49.067852 13561 raft_consensus.cc:1123] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.069254 16269 tablet_copy_client.cc:420] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:49.069329 16259 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:49.069584 16259 tablet_bootstrap.cc:439] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: Bootstrap complete. I1121 10:21:49.069677 16259 ts_tablet_manager.cc:1104] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s I1121 10:21:49.069866 16259 raft_consensus.cc:337] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d [term 2 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:49.069895 16259 raft_consensus.cc:702] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2c3332f15c014e68bed03f69d8c6287d, State: Initialized, Role: LEARNER I1121 10:21:49.069954 16259 consensus_queue.cc:231] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: NON_VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: true } } I1121 10:21:49.070094 16259 ts_tablet_manager.cc:1132] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:21:49.070103 16269 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap starting. I1121 10:21:49.071102 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 2c3332f15c014e68bed03f69d8c6287d-5f7d9348eaaa4477b9dd581c28e2d5e3 received from {username='slave'} at 127.5.70.6:37621 I1121 10:21:49.071139 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 2c3332f15c014e68bed03f69d8c6287d-5f7d9348eaaa4477b9dd581c28e2d5e3 on tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 with peer 2c3332f15c014e68bed03f69d8c6287d I1121 10:21:49.073529 16269 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:49.073691 16269 tablet_bootstrap.cc:439] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: Bootstrap complete. I1121 10:21:49.073809 16269 ts_tablet_manager.cc:1104] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I1121 10:21:49.073961 16269 raft_consensus.cc:337] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:49.073984 16269 raft_consensus.cc:702] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Initialized, Role: LEARNER I1121 10:21:49.074033 16269 consensus_queue.cc:231] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: NON_VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: true } } I1121 10:21:49.074113 16269 ts_tablet_manager.cc:1132] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:49.074486 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-7b34e50304ff46e59622de2e979ac7b8 received from {username='slave'} at 127.5.70.8:52727 I1121 10:21:49.074513 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 97a8ceca326a4f45b15a5b9c4b4857fd-7b34e50304ff46e59622de2e979ac7b8 on tablet 7b34e50304ff46e59622de2e979ac7b8 with peer 97a8ceca326a4f45b15a5b9c4b4857fd I1121 10:21:49.079468 16271 tablet_copy_client.cc:420] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:49.080145 13561 raft_consensus.cc:1123] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.080230 16271 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap starting. I1121 10:21:49.083498 16271 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:49.083703 16271 tablet_bootstrap.cc:439] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: Bootstrap complete. I1121 10:21:49.083786 16271 ts_tablet_manager.cc:1104] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s I1121 10:21:49.083911 16271 raft_consensus.cc:337] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:49.083932 16271 raft_consensus.cc:702] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fed6a935d10042b48cdf1f30824f6c57, State: Initialized, Role: LEARNER I1121 10:21:49.083981 16271 consensus_queue.cc:231] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } I1121 10:21:49.084067 16271 ts_tablet_manager.cc:1132] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57: Time spent starting tablet: real 0.000s user 0.000s sys 0.003s I1121 10:21:49.084452 10845 tablet_copy_service.cc:325] P cd693d36f0014dc3a7c3f26196124794: Request end of tablet copy session fed6a935d10042b48cdf1f30824f6c57-3e0bc1c59a1b4e2c9e761a6de8085226 received from {username='slave'} at 127.5.70.5:38887 I1121 10:21:49.084480 10845 tablet_copy_service.cc:413] P cd693d36f0014dc3a7c3f26196124794: ending tablet copy session fed6a935d10042b48cdf1f30824f6c57-3e0bc1c59a1b4e2c9e761a6de8085226 on tablet 3e0bc1c59a1b4e2c9e761a6de8085226 with peer fed6a935d10042b48cdf1f30824f6c57 I1121 10:21:49.088076 13561 raft_consensus.cc:1123] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.105443 12637 raft_consensus.cc:1123] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[] I1121 10:21:49.106977 16272 tablet_copy_client.cc:420] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:49.107722 16272 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:49.110970 16272 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:49.111129 16272 tablet_bootstrap.cc:439] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:49.111202 16272 ts_tablet_manager.cc:1104] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s I1121 10:21:49.111328 16272 raft_consensus.cc:337] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:49.111347 16272 raft_consensus.cc:702] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:49.111382 16272 consensus_queue.cc:231] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:49.111452 16272 ts_tablet_manager.cc:1132] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:49.111774 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-6765d5e95b8f45d4a8f0bb9e5d76b9f0 received from {username='slave'} at 127.5.70.9:41607 I1121 10:21:49.111807 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-6765d5e95b8f45d4a8f0bb9e5d76b9f0 on tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:49.124310 16273 tablet_copy_client.cc:420] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:49.125072 16273 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:49.128336 16273 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:49.128501 16273 tablet_bootstrap.cc:439] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:49.128628 16273 ts_tablet_manager.cc:1104] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s I1121 10:21:49.128772 16273 raft_consensus.cc:337] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:49.128798 16273 raft_consensus.cc:702] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:49.128851 16273 consensus_queue.cc:231] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:49.128932 16273 ts_tablet_manager.cc:1132] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:49.129175 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-114dbe156d634fca957db3164f8afabb received from {username='slave'} at 127.5.70.7:43411 I1121 10:21:49.129211 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-114dbe156d634fca957db3164f8afabb on tablet 114dbe156d634fca957db3164f8afabb with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:49.155498 12513 raft_consensus.cc:1123] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.222813 12265 raft_consensus.cc:1123] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.260172 12389 raft_consensus.cc:1123] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[] I1121 10:21:49.267326 12637 raft_consensus.cc:1123] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.288075 13561 raft_consensus.cc:1123] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.366775 12513 raft_consensus.cc:1123] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.417074 13561 raft_consensus.cc:1123] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.446091 12636 raft_consensus.cc:1123] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.453411 12265 raft_consensus.cc:1123] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.493013 16347 raft_consensus.cc:971] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:49.493283 16347 consensus_queue.cc:208] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:49.493778 13561 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.493856 12265 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.494002 16792 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.494103 16792 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.494119 10953 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.494514 16799 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.495251 16792 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.495460 13561 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.495497 10953 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.496482 12265 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.496670 10705 catalog_manager.cc:4143] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:49.501696 11080 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53638 I1121 10:21:49.501739 11080 raft_consensus.cc:581] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:49.540967 16561 raft_consensus.cc:971] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:49.541182 16561 consensus_queue.cc:208] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:49.541682 10816 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.541744 13561 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.541832 12389 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.542084 16561 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.542286 16873 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.542780 16876 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.543098 16561 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.543231 12389 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.544317 10816 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.544335 10701 catalog_manager.cc:4143] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:49.546401 13561 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.546790 11205 consensus_queue.cc:208] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:49.547255 13560 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.547415 12389 raft_consensus.cc:1181] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.547619 16876 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.547737 16873 consensus_queue.cc:984] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.548297 16876 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.548460 13560 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.548555 12389 raft_consensus.cc:2801] T 172c695aadee4de6ad0a5a6508b290f6 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.549089 10693 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 172c695aadee4de6ad0a5a6508b290f6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:49.549245 10710 catalog_manager.cc:4143] T 172c695aadee4de6ad0a5a6508b290f6 P 5457a73f9ecb40ee9889513a8412937b reported cstate change: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:49.549836 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet 172c695aadee4de6ad0a5a6508b290f6 with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:49.550014 16885 tablet_replica.cc:282] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:49.550072 16885 raft_consensus.cc:2156] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:49.550117 16885 raft_consensus.cc:2183] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:49.550362 16885 ts_tablet_manager.cc:1457] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:49.551766 16885 ts_tablet_manager.cc:1470] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:49.551801 16885 log.cc:1153] T 172c695aadee4de6ad0a5a6508b290f6 P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/172c695aadee4de6ad0a5a6508b290f6 I1121 10:21:49.552162 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet 172c695aadee4de6ad0a5a6508b290f6 (table rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]) successfully deleted I1121 10:21:49.553985 12265 raft_consensus.cc:1123] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:49.559121 16344 raft_consensus.cc:971] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:49.559365 16344 consensus_queue.cc:208] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:49.559857 12389 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.559965 11205 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.560070 15876 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.560173 15876 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.560308 13560 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.560734 15876 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.561107 15876 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.561278 11205 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.561534 13560 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.562379 12389 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.562441 10701 catalog_manager.cc:4143] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:49.563352 10953 consensus_queue.cc:208] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:49.563689 13560 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.563693 12389 raft_consensus.cc:1181] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.563905 15876 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.563977 16888 consensus_queue.cc:984] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.564622 16888 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.564824 13560 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.564851 12389 raft_consensus.cc:2801] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.565555 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 69c25581de6e4945a167245a4dcbbfe0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:49.565696 10710 catalog_manager.cc:4143] T 69c25581de6e4945a167245a4dcbbfe0 P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 8 to 9, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.566334 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet 69c25581de6e4945a167245a4dcbbfe0 with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:49.566550 16892 tablet_replica.cc:282] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:49.566599 16892 raft_consensus.cc:2156] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:49.566627 16892 raft_consensus.cc:2183] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:49.566872 16892 ts_tablet_manager.cc:1457] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:49.568305 16892 ts_tablet_manager.cc:1470] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:49.568338 16892 log.cc:1153] T 69c25581de6e4945a167245a4dcbbfe0 P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/69c25581de6e4945a167245a4dcbbfe0 I1121 10:21:49.568760 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet 69c25581de6e4945a167245a4dcbbfe0 (table rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]) successfully deleted I1121 10:21:49.574501 16887 raft_consensus.cc:971] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:49.574707 16887 consensus_queue.cc:208] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:49.575373 12513 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.575469 10816 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.575490 12389 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.575585 16887 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.575687 15876 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.575831 16887 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.576436 16887 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.576671 10815 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.576736 12389 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.577652 10701 catalog_manager.cc:4143] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.578186 12513 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.578630 10953 consensus_queue.cc:208] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:49.579164 12513 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.579233 12389 raft_consensus.cc:1181] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.579411 15876 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.579488 16888 consensus_queue.cc:984] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.580123 16887 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.580255 12389 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.580289 12513 raft_consensus.cc:2801] T 114dbe156d634fca957db3164f8afabb P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.580935 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 114dbe156d634fca957db3164f8afabb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:49.581115 10710 catalog_manager.cc:4143] T 114dbe156d634fca957db3164f8afabb P 2c3332f15c014e68bed03f69d8c6287d reported cstate change: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.581851 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet 114dbe156d634fca957db3164f8afabb with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:49.581925 16885 tablet_replica.cc:282] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:49.581984 16885 raft_consensus.cc:2156] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:49.582008 16885 raft_consensus.cc:2183] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:49.582247 16885 ts_tablet_manager.cc:1457] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:49.583616 16885 ts_tablet_manager.cc:1470] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:49.583649 16885 log.cc:1153] T 114dbe156d634fca957db3164f8afabb P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/114dbe156d634fca957db3164f8afabb I1121 10:21:49.584015 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet 114dbe156d634fca957db3164f8afabb (table rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]) successfully deleted I1121 10:21:49.652369 16799 raft_consensus.cc:971] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:49.652735 16799 consensus_queue.cc:208] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:49.653295 10815 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:49.653326 12636 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:49.653493 12389 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:49.653519 16347 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.653635 16799 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.653694 16347 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.654587 16792 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.654738 10815 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.654781 12636 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.655879 12389 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.655886 10701 catalog_manager.cc:4143] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 8 to 9, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.657052 11080 consensus_queue.cc:208] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:49.657413 12389 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:49.657541 12636 raft_consensus.cc:1181] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:49.657604 16347 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:49.657721 16347 consensus_queue.cc:984] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:49.658215 16797 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.658355 12389 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.659023 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 1fa5ac03774f4944815d262fcdffa2f6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:49.659224 10710 catalog_manager.cc:4143] T 1fa5ac03774f4944815d262fcdffa2f6 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 9 to 10, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 2 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:49.659651 12636 raft_consensus.cc:2801] T 1fa5ac03774f4944815d262fcdffa2f6 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.659942 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet 1fa5ac03774f4944815d262fcdffa2f6 with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:49.660012 16885 tablet_replica.cc:282] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:49.660094 16885 raft_consensus.cc:2156] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:49.660127 16885 raft_consensus.cc:2183] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:49.660470 16885 ts_tablet_manager.cc:1457] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:49.661852 16885 ts_tablet_manager.cc:1470] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9 I1121 10:21:49.661890 16885 log.cc:1153] T 1fa5ac03774f4944815d262fcdffa2f6 P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/1fa5ac03774f4944815d262fcdffa2f6 I1121 10:21:49.662194 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet 1fa5ac03774f4944815d262fcdffa2f6 (table rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]) successfully deleted I1121 10:21:49.670399 16347 raft_consensus.cc:971] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:49.670675 16347 consensus_queue.cc:208] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:49.671180 13560 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.671231 12513 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.671368 10815 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.671530 16792 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.671607 16347 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.671733 16799 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.672464 16792 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.672633 12512 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.673558 10815 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.673585 10710 catalog_manager.cc:4143] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:49.673653 13560 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.674669 11080 consensus_queue.cc:208] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:49.675026 13560 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.675180 12512 raft_consensus.cc:1181] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.675418 16799 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s W1121 10:21:49.675420 10690 catalog_manager.cc:3695] ChangeConfig:REMOVE_PEER RPC for tablet a40d6cf45df24788adc85f238c9bcc7e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER failed with leader ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } }. Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:49.675516 16347 consensus_queue.cc:984] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.676082 16797 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.676228 13560 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.676704 12512 raft_consensus.cc:2801] T a40d6cf45df24788adc85f238c9bcc7e P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.677253 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet a40d6cf45df24788adc85f238c9bcc7e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:49.677299 10710 catalog_manager.cc:4143] T a40d6cf45df24788adc85f238c9bcc7e P 80d2057cd4814fd3bfe268270d1abdd4 reported cstate change: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:49.678020 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet a40d6cf45df24788adc85f238c9bcc7e with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:49.678099 16885 tablet_replica.cc:282] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:49.678151 16885 raft_consensus.cc:2156] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:49.678171 16885 raft_consensus.cc:2183] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:49.678416 16885 ts_tablet_manager.cc:1457] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:49.679585 16885 ts_tablet_manager.cc:1470] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:49.679620 16885 log.cc:1153] T a40d6cf45df24788adc85f238c9bcc7e P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/a40d6cf45df24788adc85f238c9bcc7e I1121 10:21:49.679894 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet a40d6cf45df24788adc85f238c9bcc7e (table rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]) successfully deleted I1121 10:21:49.738528 10690 catalog_manager.cc:3712] ChangeConfig:REMOVE_PEER RPC for tablet a40d6cf45df24788adc85f238c9bcc7e with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8 I1121 10:21:49.752399 11080 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53668 I1121 10:21:49.752463 11080 raft_consensus.cc:581] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:49.776677 16791 raft_consensus.cc:971] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:49.776962 16791 consensus_queue.cc:208] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:49.777397 12512 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce [term 1 LEARNER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.777421 12265 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.777531 11205 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.777634 16791 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.777730 16956 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.777791 16791 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.778692 16957 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.778785 12512 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.778882 12265 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.779965 10701 catalog_manager.cc:4143] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce reported cstate change: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.780036 11205 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.781883 10815 consensus_queue.cc:208] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:49.782325 12265 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.782335 12512 raft_consensus.cc:1181] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.782512 16791 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.782567 16956 consensus_queue.cc:984] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.783149 16791 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.783273 12512 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.783437 12265 raft_consensus.cc:2801] T c7de6e746a8246f89d8b65b59803c3af P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.784018 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet c7de6e746a8246f89d8b65b59803c3af with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:49.784211 10701 catalog_manager.cc:4143] T c7de6e746a8246f89d8b65b59803c3af P 77830a7609a549e0a1c76586d0e301ce reported cstate change: config changed from index 8 to 9, VOTER 5457a73f9ecb40ee9889513a8412937b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:49.785219 11180 tablet_service.cc:1067] Processing DeleteTablet for tablet c7de6e746a8246f89d8b65b59803c3af with delete_type TABLET_DATA_TOMBSTONED (TS 5457a73f9ecb40ee9889513a8412937b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39700 I1121 10:21:49.785346 16892 tablet_replica.cc:282] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b: stopping tablet replica I1121 10:21:49.785388 16892 raft_consensus.cc:2156] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:49.785403 16892 raft_consensus.cc:2183] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:49.785781 16892 ts_tablet_manager.cc:1457] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:49.787343 16892 ts_tablet_manager.cc:1470] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:49.787381 16892 log.cc:1153] T c7de6e746a8246f89d8b65b59803c3af P 5457a73f9ecb40ee9889513a8412937b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/c7de6e746a8246f89d8b65b59803c3af I1121 10:21:49.787734 10693 catalog_manager.cc:3512] TS 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207): tablet c7de6e746a8246f89d8b65b59803c3af (table rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]) successfully deleted I1121 10:21:49.801826 11205 consensus_queue.cc:208] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:49.802316 10815 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:49.802315 12636 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:49.802528 16561 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s W1121 10:21:49.802603 11136 consensus_peers.cc:473] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df2330a22c8142ada6d056401d0a064d. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:49.802722 16876 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:49.803436 16873 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:49.803578 12636 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:49.803678 10815 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:49.804757 10705 catalog_manager.cc:4143] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:49.808933 10815 consensus_queue.cc:208] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:49.809481 11080 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:49.809681 16957 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:49.809689 12636 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:49.810009 16957 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:49.810581 16346 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:49.810775 11080 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:49.811668 10701 catalog_manager.cc:4143] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:49.811858 12636 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } W1121 10:21:49.812855 10763 consensus_peers.cc:473] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 702635e315b940138670eeb5ae6ace1b. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:49.816576 15876 raft_consensus.cc:971] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 2c3332f15c014e68bed03f69d8c6287d to VOTER I1121 10:21:49.816793 15876 consensus_queue.cc:208] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:49.816965 10953 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:49.817170 12389 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d [term 2 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:49.817308 11080 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:49.817406 11080 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:49.817414 12265 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:49.817600 16888 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.817610 16887 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:49.817703 16344 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.817798 12636 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:49.817904 16887 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s W1121 10:21:49.817956 10888 consensus_peers.cc:473] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ecac46041da45aba6e82df02354fe89. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:49.817981 16887 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:49.818470 16887 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:49.818641 11080 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:49.818699 15876 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:49.819641 10710 catalog_manager.cc:4143] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 8 to 9, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:49.819713 11080 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:49.819878 12636 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:49.819973 16347 raft_consensus.cc:971] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:49.820143 16347 consensus_queue.cc:208] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:49.820482 12265 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:49.820588 12264 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.820639 12637 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.820778 10815 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.820842 16792 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.821341 16792 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.821499 10690 catalog_manager.cc:3712] ChangeConfig:REMOVE_PEER RPC for tablet c7de6e746a8246f89d8b65b59803c3af with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8 I1121 10:21:49.821676 16792 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.821887 10701 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:49.822496 12389 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:49.823648 10953 consensus_queue.cc:208] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } I1121 10:21:49.823742 16792 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:49.824223 12389 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:49.824389 12264 raft_consensus.cc:1181] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:49.824506 15876 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:49.824895 12636 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:49.825080 10702 catalog_manager.cc:4143] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:49.825095 12264 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:49.825614 15876 consensus_queue.cc:984] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:49.826189 15876 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:49.826467 11080 consensus_queue.cc:208] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:49.826803 12265 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.827018 17019 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.827332 10702 catalog_manager.cc:4143] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 9 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 2 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:49.827353 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:49.827387 12636 raft_consensus.cc:1181] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:49.827594 12264 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:49.827852 12389 raft_consensus.cc:2801] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } } I1121 10:21:49.827909 16347 consensus_queue.cc:984] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:49.828351 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:49.828691 16347 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:49.828781 12636 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:49.829162 12265 raft_consensus.cc:2801] T 69ac58ad89e447a494fc407037d27f8d P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:49.829733 10710 catalog_manager.cc:4143] T 69ac58ad89e447a494fc407037d27f8d P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:49.829857 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 69ac58ad89e447a494fc407037d27f8d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:49.830471 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet 69ac58ad89e447a494fc407037d27f8d with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:49.830541 16885 tablet_replica.cc:282] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:49.830549 17028 tablet_replica.cc:282] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:49.830585 16885 raft_consensus.cc:2156] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:49.830591 17028 raft_consensus.cc:2156] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:49.830596 16885 pending_rounds.cc:64] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: Trying to abort 1 pending transactions. I1121 10:21:49.830605 17028 raft_consensus.cc:2183] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:49.830718 16885 pending_rounds.cc:70] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: Aborting transaction as it isn't in flight: id { term: 1 index: 8 } timestamp: 6448462683422781440 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "69ac58ad89e447a494fc407037d27f8d" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: NON_VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } } I1121 10:21:49.830734 16885 raft_consensus.cc:2763] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Transaction aborted I1121 10:21:49.830746 16885 raft_consensus.cc:2736] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Transaction aborted I1121 10:21:49.830752 16885 raft_consensus.cc:2183] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:49.830894 17028 ts_tablet_manager.cc:1457] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:49.831130 16885 ts_tablet_manager.cc:1457] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:49.832413 17028 ts_tablet_manager.cc:1470] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9 I1121 10:21:49.832454 17028 log.cc:1153] T 5f7d9348eaaa4477b9dd581c28e2d5e3 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/5f7d9348eaaa4477b9dd581c28e2d5e3 I1121 10:21:49.832568 16885 ts_tablet_manager.cc:1470] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:49.832607 16885 log.cc:1153] T 69ac58ad89e447a494fc407037d27f8d P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/69ac58ad89e447a494fc407037d27f8d I1121 10:21:49.832917 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 (table rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]) successfully deleted I1121 10:21:49.832955 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet 69ac58ad89e447a494fc407037d27f8d (table rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]) successfully deleted I1121 10:21:49.833493 11070 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53704 I1121 10:21:49.833520 11070 raft_consensus.cc:581] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:49.850478 16799 raft_consensus.cc:971] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:49.850760 16799 consensus_queue.cc:208] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:49.851214 12512 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.851322 11205 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.851433 16347 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.851485 12636 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:49.851516 16347 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.851683 16347 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:49.852710 16347 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.852959 11205 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.853852 12512 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.853927 10710 catalog_manager.cc:4143] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:49.854095 12636 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:49.859585 10690 catalog_manager.cc:3712] ChangeConfig:REMOVE_PEER RPC for tablet 69ac58ad89e447a494fc407037d27f8d with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8 I1121 10:21:49.861492 11070 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53720 I1121 10:21:49.861527 11070 raft_consensus.cc:581] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:49.876919 11070 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53730 I1121 10:21:49.876958 11070 raft_consensus.cc:581] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:49.895851 11070 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53742 I1121 10:21:49.895891 11070 raft_consensus.cc:581] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:50.005297 16797 raft_consensus.cc:971] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:50.005501 16797 consensus_queue.cc:208] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:50.006039 12265 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.006064 12389 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.006243 10953 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.006279 16792 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.006371 17019 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.006456 16792 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.007273 16792 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.007441 12265 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.008566 10710 catalog_manager.cc:4143] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57 reported cstate change: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.008643 10953 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.008791 12389 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.009987 11070 consensus_queue.cc:208] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:50.010301 12265 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:50.010515 12389 raft_consensus.cc:1181] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:50.010679 17019 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.010754 16797 consensus_queue.cc:984] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.011358 16797 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.011538 12389 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.012238 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 635b29a1f41e418faffe3677886048e5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:50.012553 12265 raft_consensus.cc:2801] T 635b29a1f41e418faffe3677886048e5 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.012614 10702 catalog_manager.cc:4143] T 635b29a1f41e418faffe3677886048e5 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:50.012869 16792 raft_consensus.cc:971] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:50.013005 16792 consensus_queue.cc:208] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:50.013438 10953 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.013643 16799 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.013793 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 635b29a1f41e418faffe3677886048e5 with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:50.013942 17153 tablet_replica.cc:282] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:50.014029 17153 raft_consensus.cc:2156] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:50.014060 17153 raft_consensus.cc:2183] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:50.014089 13560 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.014097 12389 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.014346 16799 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.014426 16347 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.014648 17153 ts_tablet_manager.cc:1457] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:50.015138 16347 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:50.015362 13560 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:50.016090 10953 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:50.016141 10702 catalog_manager.cc:4143] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:50.016474 17153 ts_tablet_manager.cc:1470] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:50.016549 17153 log.cc:1153] T 635b29a1f41e418faffe3677886048e5 P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/635b29a1f41e418faffe3677886048e5 I1121 10:21:50.017252 12389 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:50.017272 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 635b29a1f41e418faffe3677886048e5 (table rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]) successfully deleted I1121 10:21:50.029683 17019 raft_consensus.cc:900] Instructing follower af0d02c642ed4eab8d971d9f05d32044 to start an election I1121 10:21:50.029724 17019 raft_consensus.cc:988] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Signalling peer af0d02c642ed4eab8d971d9f05d32044to start an election I1121 10:21:50.029951 10953 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" from {username='slave'} at 127.5.70.3:46687 I1121 10:21:50.029996 10953 raft_consensus.cc:469] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:21:50.030010 10953 raft_consensus.cc:2905] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.030555 10953 raft_consensus.cc:491] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:50.030643 10953 leader_election.cc:296] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 2 election: Requested vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757), 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291) I1121 10:21:50.030895 13560 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "80d2057cd4814fd3bfe268270d1abdd4" I1121 10:21:50.030892 12265 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "fed6a935d10042b48cdf1f30824f6c57" I1121 10:21:50.030930 12265 raft_consensus.cc:2905] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.030943 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fa85115a8bbd45dbb7ce1f18b923022a" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:50.030932 13560 raft_consensus.cc:2905] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.030961 11080 raft_consensus.cc:2900] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:21:50.030966 11080 raft_consensus.cc:702] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:50.031049 11080 consensus_queue.cc:231] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:50.031077 11080 raft_consensus.cc:2905] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.031546 11080 raft_consensus.cc:2359] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 2. I1121 10:21:50.031658 13560 raft_consensus.cc:2359] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 2. I1121 10:21:50.031661 12265 raft_consensus.cc:2359] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 2. I1121 10:21:50.031810 10888 leader_election.cc:310] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 80d2057cd4814fd3bfe268270d1abdd4, af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:50.031883 17016 raft_consensus.cc:2653] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:21:50.031920 17016 raft_consensus.cc:664] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 2 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:50.032019 17016 consensus_queue.cc:208] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:50.032579 10702 catalog_manager.cc:4143] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 1 to 2, leader changed from ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 2 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:50.049505 17016 raft_consensus.cc:971] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 97a8ceca326a4f45b15a5b9c4b4857fd to VOTER I1121 10:21:50.049731 17016 consensus_queue.cc:208] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:50.050190 12636 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.050241 10815 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.050248 12265 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.050439 17017 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.050529 16888 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.050662 16887 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.051528 16888 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:50.051656 12636 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:50.051729 10815 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:50.052768 10702 catalog_manager.cc:4143] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:50.053126 12265 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:50.062657 10690 catalog_manager.cc:3712] ChangeConfig:REMOVE_PEER RPC for tablet 635b29a1f41e418faffe3677886048e5 with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8 I1121 10:21:50.100828 16346 raft_consensus.cc:971] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794: attempting to promote NON_VOTER fed6a935d10042b48cdf1f30824f6c57 to VOTER I1121 10:21:50.101048 16346 consensus_queue.cc:208] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:50.101505 12265 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57 [term 1 LEARNER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.101549 10953 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.101578 12636 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:50.101756 16956 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.101836 16791 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.101913 16957 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:50.102766 16956 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.102934 12636 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.103263 12265 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.104071 10705 catalog_manager.cc:4143] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.104147 10953 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.105048 10815 consensus_queue.cc:208] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } I1121 10:21:50.105438 12265 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:50.105628 12636 raft_consensus.cc:1181] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:50.105646 16791 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.105845 16791 consensus_queue.cc:984] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.106371 16956 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.106513 12636 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.106513 12265 raft_consensus.cc:2801] T 3e0bc1c59a1b4e2c9e761a6de8085226 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } } I1121 10:21:50.107389 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 3e0bc1c59a1b4e2c9e761a6de8085226 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:50.107488 10705 catalog_manager.cc:4143] T 3e0bc1c59a1b4e2c9e761a6de8085226 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:50.108199 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 3e0bc1c59a1b4e2c9e761a6de8085226 with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:50.108278 17153 tablet_replica.cc:282] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:50.108327 17153 raft_consensus.cc:2156] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:50.108347 17153 raft_consensus.cc:2183] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:50.108616 17153 ts_tablet_manager.cc:1457] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:50.109874 17153 ts_tablet_manager.cc:1470] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:50.109917 17153 log.cc:1153] T 3e0bc1c59a1b4e2c9e761a6de8085226 P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/3e0bc1c59a1b4e2c9e761a6de8085226 I1121 10:21:50.110268 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 3e0bc1c59a1b4e2c9e761a6de8085226 (table rebalance_test_table_7 [id=d9d752ddf70440a2bb9ed3eb94446021]) successfully deleted I1121 10:21:50.119395 11080 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53756 I1121 10:21:50.119432 11080 raft_consensus.cc:581] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:50.129305 10953 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" mode: GRACEFUL from {username='slave'} at 127.0.0.1:50056 I1121 10:21:50.129353 10953 raft_consensus.cc:581] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:50.135191 11080 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53766 I1121 10:21:50.135229 11080 raft_consensus.cc:581] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:50.154036 11080 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53778 I1121 10:21:50.154075 11080 raft_consensus.cc:581] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:50.162894 10953 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" mode: GRACEFUL from {username='slave'} at 127.0.0.1:50078 I1121 10:21:50.162935 10953 raft_consensus.cc:581] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:50.168953 11080 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53788 I1121 10:21:50.168993 11080 raft_consensus.cc:581] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:50.238557 17275 ts_tablet_manager.cc:721] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:50.238608 17275 tablet_copy_client.cc:243] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:43035 I1121 10:21:50.241492 10845 tablet_copy_service.cc:135] P cd693d36f0014dc3a7c3f26196124794: Received BeginTabletCopySession request for tablet 702635e315b940138670eeb5ae6ace1b from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:50887) I1121 10:21:50.241521 10845 tablet_copy_service.cc:156] P cd693d36f0014dc3a7c3f26196124794: Beginning new tablet copy session on tablet 702635e315b940138670eeb5ae6ace1b from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:50887: session id = 80d2057cd4814fd3bfe268270d1abdd4-702635e315b940138670eeb5ae6ace1b I1121 10:21:50.242012 10845 tablet_copy_source_session.cc:204] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:50.242512 17275 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 702635e315b940138670eeb5ae6ace1b. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:50.242908 17019 raft_consensus.cc:900] Instructing follower 97a8ceca326a4f45b15a5b9c4b4857fd to start an election I1121 10:21:50.242962 17019 raft_consensus.cc:988] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Signalling peer 97a8ceca326a4f45b15a5b9c4b4857fdto start an election I1121 10:21:50.243269 12636 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" dest_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" from {username='slave'} at 127.5.70.3:59439 I1121 10:21:50.243309 12636 raft_consensus.cc:469] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:21:50.243328 12636 raft_consensus.cc:2905] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.243861 17275 tablet_copy_client.cc:581] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:50.243955 17275 tablet_copy_client.cc:544] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:50.244060 12636 raft_consensus.cc:491] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:50.244436 12636 leader_election.cc:296] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [CANDIDATE]: Term 2 election: Requested vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193) I1121 10:21:50.244670 11205 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:50.244714 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:50.244735 11080 raft_consensus.cc:2900] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:21:50.244743 11080 raft_consensus.cc:702] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:50.244730 11205 raft_consensus.cc:2905] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.244823 11080 consensus_queue.cc:231] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:50.244868 11080 raft_consensus.cc:2905] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.245568 11080 raft_consensus.cc:2359] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:50.245568 11205 raft_consensus.cc:2359] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:50.245666 17275 tablet_copy_client.cc:755] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:50.245759 12567 leader_election.cc:310] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, 97a8ceca326a4f45b15a5b9c4b4857fd, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:50.245857 16638 raft_consensus.cc:2653] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:21:50.245968 16638 raft_consensus.cc:664] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Becoming Leader. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Running, Role: LEADER I1121 10:21:50.246085 16638 consensus_queue.cc:208] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:50.246762 10702 catalog_manager.cc:4143] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: term changed from 1 to 2, leader changed from ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) to 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8). New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:50.248540 12512 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "7b34e50304ff46e59622de2e979ac7b8" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "77830a7609a549e0a1c76586d0e301ce" I1121 10:21:50.248580 12512 raft_consensus.cc:2905] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.249100 12512 raft_consensus.cc:2359] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:50.339550 17280 ts_tablet_manager.cc:721] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797) I1121 10:21:50.339604 17280 tablet_copy_client.cc:243] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.2:43797 I1121 10:21:50.339901 10977 tablet_copy_service.cc:135] P af0d02c642ed4eab8d971d9f05d32044: Received BeginTabletCopySession request for tablet 4ecac46041da45aba6e82df02354fe89 from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:35497) I1121 10:21:50.339931 10977 tablet_copy_service.cc:156] P af0d02c642ed4eab8d971d9f05d32044: Beginning new tablet copy session on tablet 4ecac46041da45aba6e82df02354fe89 from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:35497: session id = 80d2057cd4814fd3bfe268270d1abdd4-4ecac46041da45aba6e82df02354fe89 I1121 10:21:50.340348 10977 tablet_copy_source_session.cc:204] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:50.340742 17280 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ecac46041da45aba6e82df02354fe89. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:50.341727 17280 tablet_copy_client.cc:581] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:50.341825 17280 tablet_copy_client.cc:544] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:50.342340 17280 tablet_copy_client.cc:755] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:50.385150 11080 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:53798 I1121 10:21:50.385185 11080 raft_consensus.cc:581] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:50.394490 10953 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" mode: GRACEFUL from {username='slave'} at 127.0.0.1:50098 I1121 10:21:50.394526 10953 raft_consensus.cc:581] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:50.405325 17327 ts_tablet_manager.cc:721] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207) I1121 10:21:50.405375 17327 tablet_copy_client.cc:243] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:37207 I1121 10:21:50.405699 11218 tablet_copy_service.cc:135] P 5457a73f9ecb40ee9889513a8412937b: Received BeginTabletCopySession request for tablet df2330a22c8142ada6d056401d0a064d from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:44295) I1121 10:21:50.405735 11218 tablet_copy_service.cc:156] P 5457a73f9ecb40ee9889513a8412937b: Beginning new tablet copy session on tablet df2330a22c8142ada6d056401d0a064d from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:44295: session id = 80d2057cd4814fd3bfe268270d1abdd4-df2330a22c8142ada6d056401d0a064d I1121 10:21:50.406199 11218 tablet_copy_source_session.cc:204] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:50.406594 17327 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet df2330a22c8142ada6d056401d0a064d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:50.407621 17327 tablet_copy_client.cc:581] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:50.407696 17327 tablet_copy_client.cc:544] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:50.408311 17327 tablet_copy_client.cc:755] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:50.433499 12265 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:50.433779 17017 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.455060 11080 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:50.455343 17017 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.456832 13560 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:50.457193 17016 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.507190 17016 raft_consensus.cc:900] Instructing follower 97a8ceca326a4f45b15a5b9c4b4857fd to start an election I1121 10:21:50.507251 17016 raft_consensus.cc:988] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Signalling peer 97a8ceca326a4f45b15a5b9c4b4857fdto start an election I1121 10:21:50.507550 12636 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" dest_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" from {username='slave'} at 127.5.70.2:45103 I1121 10:21:50.507591 12636 raft_consensus.cc:469] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:21:50.507604 12636 raft_consensus.cc:2905] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.508332 12636 raft_consensus.cc:491] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:50.508436 12636 leader_election.cc:296] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [CANDIDATE]: Term 2 election: Requested vote from peers af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), fed6a935d10042b48cdf1f30824f6c57 (127.5.70.5:35757) I1121 10:21:50.508667 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:50.508760 10953 raft_consensus.cc:2900] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:21:50.508785 10953 raft_consensus.cc:702] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:50.508831 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:50.508868 10815 raft_consensus.cc:2905] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.508874 10953 consensus_queue.cc:231] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:50.508919 10953 raft_consensus.cc:2905] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.509572 10815 raft_consensus.cc:2359] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:50.509702 10953 raft_consensus.cc:2359] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:50.509889 12569 leader_election.cc:310] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 97a8ceca326a4f45b15a5b9c4b4857fd, af0d02c642ed4eab8d971d9f05d32044, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:50.509955 16638 raft_consensus.cc:2653] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:21:50.509987 16638 raft_consensus.cc:664] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Becoming Leader. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Running, Role: LEADER I1121 10:21:50.510059 16638 consensus_queue.cc:208] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:50.510617 10710 catalog_manager.cc:4143] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: term changed from 1 to 2, leader changed from af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) to 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8). New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:50.511808 12265 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "25b55ff241d8442897ea97523a2cd6fd" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "fed6a935d10042b48cdf1f30824f6c57" I1121 10:21:50.511840 12265 raft_consensus.cc:2905] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.512409 12265 raft_consensus.cc:2359] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:50.545408 16799 raft_consensus.cc:900] Instructing follower af0d02c642ed4eab8d971d9f05d32044 to start an election I1121 10:21:50.545460 16799 raft_consensus.cc:988] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Signalling peer af0d02c642ed4eab8d971d9f05d32044to start an election I1121 10:21:50.545660 10953 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" from {username='slave'} at 127.5.70.3:46687 I1121 10:21:50.545718 10953 raft_consensus.cc:469] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:21:50.545729 10953 raft_consensus.cc:2905] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.546731 10953 raft_consensus.cc:491] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:50.546830 10953 leader_election.cc:296] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 2 election: Requested vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 2c3332f15c014e68bed03f69d8c6287d (127.5.70.6:37787), 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291) I1121 10:21:50.547065 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:50.547092 13560 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "80d2057cd4814fd3bfe268270d1abdd4" I1121 10:21:50.547103 11080 raft_consensus.cc:2900] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:21:50.547116 11080 raft_consensus.cc:702] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:50.547122 13560 raft_consensus.cc:2905] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.547204 11080 consensus_queue.cc:231] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:50.547309 11080 raft_consensus.cc:2905] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.547389 12389 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "6765d5e95b8f45d4a8f0bb9e5d76b9f0" candidate_uuid: "af0d02c642ed4eab8d971d9f05d32044" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2c3332f15c014e68bed03f69d8c6287d" I1121 10:21:50.547432 12389 raft_consensus.cc:2905] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:50.547843 13560 raft_consensus.cc:2359] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 2. I1121 10:21:50.547987 11080 raft_consensus.cc:2359] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 2. I1121 10:21:50.548115 10887 leader_election.cc:310] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 80d2057cd4814fd3bfe268270d1abdd4, af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:50.548208 17016 raft_consensus.cc:2653] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:21:50.548246 12389 raft_consensus.cc:2359] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af0d02c642ed4eab8d971d9f05d32044 in term 2. I1121 10:21:50.548250 17016 raft_consensus.cc:664] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 2 LEADER]: Becoming Leader. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:50.548338 17016 consensus_queue.cc:208] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:50.549083 10710 catalog_manager.cc:4143] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 reported cstate change: term changed from 1 to 2, leader changed from ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) to af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2). New cstate: current_term: 2 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:50.690032 11205 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:50.690315 16638 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.732595 11080 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:50.732893 16638 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.734439 12512 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:50.734721 17279 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.923233 10815 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:50.923522 17279 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:50.985292 11080 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:50.985579 17016 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:51.012374 10953 consensus_queue.cc:208] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:51.012850 13560 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:51.012899 12265 raft_consensus.cc:1181] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:51.013245 16887 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:51.013317 17016 consensus_queue.cc:984] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:51.014086 17016 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P af0d02c642ed4eab8d971d9f05d32044 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:51.014248 12265 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:51.014344 13560 raft_consensus.cc:2801] T fa85115a8bbd45dbb7ce1f18b923022a P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:51.015416 10702 catalog_manager.cc:4143] T fa85115a8bbd45dbb7ce1f18b923022a P fed6a935d10042b48cdf1f30824f6c57 reported cstate change: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 2 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:51.015415 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet fa85115a8bbd45dbb7ce1f18b923022a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:51.016480 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet fa85115a8bbd45dbb7ce1f18b923022a with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:51.016651 17413 tablet_replica.cc:282] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:51.016705 17413 raft_consensus.cc:2156] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:51.016717 17413 raft_consensus.cc:2183] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:51.016988 17413 ts_tablet_manager.cc:1457] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:51.018569 17413 ts_tablet_manager.cc:1470] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9 I1121 10:21:51.018615 17413 log.cc:1153] T fa85115a8bbd45dbb7ce1f18b923022a P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/fa85115a8bbd45dbb7ce1f18b923022a I1121 10:21:51.019043 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet fa85115a8bbd45dbb7ce1f18b923022a (table rebalance_test_table_5 [id=0a41ea36010c4786b4b40cd18e5c0753]) successfully deleted I1121 10:21:51.069350 12389 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:51.069643 16887 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:51.070955 13560 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:51.071297 16888 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:51.076113 10953 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:51.076417 17279 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:51.077672 12265 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:51.077894 17279 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:51.104341 10815 consensus_queue.cc:208] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:51.104986 12265 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:51.105007 10953 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:51.105095 10763 consensus_peers.cc:473] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40e235c631e14bf8b4fe83e2953d8c87. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:51.105207 17467 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:51.105298 17375 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:51.106146 16346 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:51.106267 12265 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:51.107129 10953 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:51.108438 10702 catalog_manager.cc:4143] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:51.138334 12636 consensus_queue.cc:208] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:51.138677 12512 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:51.138720 11205 raft_consensus.cc:1181] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:51.138911 17279 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:51.138988 16638 consensus_queue.cc:984] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:51.139778 17369 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:51.139825 12512 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:51.139878 11205 raft_consensus.cc:2801] T 7b34e50304ff46e59622de2e979ac7b8 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:51.140905 10691 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 7b34e50304ff46e59622de2e979ac7b8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:51.140939 10705 catalog_manager.cc:4143] T 7b34e50304ff46e59622de2e979ac7b8 P 77830a7609a549e0a1c76586d0e301ce reported cstate change: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:51.141685 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 7b34e50304ff46e59622de2e979ac7b8 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:51.141757 17413 tablet_replica.cc:282] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:51.141803 17413 raft_consensus.cc:2156] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:51.141813 17413 raft_consensus.cc:2183] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:51.142145 17413 ts_tablet_manager.cc:1457] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:51.143532 17413 ts_tablet_manager.cc:1470] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9 I1121 10:21:51.143631 17413 log.cc:1153] T 7b34e50304ff46e59622de2e979ac7b8 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/7b34e50304ff46e59622de2e979ac7b8 I1121 10:21:51.143962 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 7b34e50304ff46e59622de2e979ac7b8 (table rebalance_test_table_1 [id=946682e3f34d4ca682a2be2971a72206]) successfully deleted I1121 10:21:51.511842 12636 consensus_queue.cc:208] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } I1121 10:21:51.512272 12265 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:51.512348 10815 raft_consensus.cc:1181] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:51.512557 17279 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:51.512665 17506 consensus_queue.cc:984] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:51.513316 17279 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:51.513401 12265 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P fed6a935d10042b48cdf1f30824f6c57 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:51.513435 10815 raft_consensus.cc:2801] T 25b55ff241d8442897ea97523a2cd6fd P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } } I1121 10:21:51.514408 10691 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 25b55ff241d8442897ea97523a2cd6fd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:51.514653 10702 catalog_manager.cc:4143] T 25b55ff241d8442897ea97523a2cd6fd P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 8 to 10, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:51.515537 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 25b55ff241d8442897ea97523a2cd6fd with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:51.515760 17573 tablet_replica.cc:282] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:51.515825 17573 raft_consensus.cc:2156] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:51.515843 17573 raft_consensus.cc:2183] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:51.516105 17573 ts_tablet_manager.cc:1457] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:51.517807 17573 ts_tablet_manager.cc:1470] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9 I1121 10:21:51.517841 17573 log.cc:1153] T 25b55ff241d8442897ea97523a2cd6fd P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/25b55ff241d8442897ea97523a2cd6fd I1121 10:21:51.518143 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 25b55ff241d8442897ea97523a2cd6fd (table rebalance_test_table_6 [id=ffdefccc99f5402f924bbd143a6538a3]) successfully deleted I1121 10:21:51.525075 10953 consensus_queue.cc:208] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:51.525514 13560 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:51.525665 12389 raft_consensus.cc:1181] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:51.525738 16888 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:51.526016 16888 consensus_queue.cc:984] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:51.526533 17016 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P af0d02c642ed4eab8d971d9f05d32044 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:51.526669 13560 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:51.526866 12389 raft_consensus.cc:2801] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 2c3332f15c014e68bed03f69d8c6287d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:51.527567 10692 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:51.527791 10705 catalog_manager.cc:4143] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P 80d2057cd4814fd3bfe268270d1abdd4 reported cstate change: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 2 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } } peers { permanent_uuid: "2c3332f15c014e68bed03f69d8c6287d" member_type: VOTER last_known_addr { host: "127.5.70.6" port: 37787 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:51.528815 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:51.528918 17413 tablet_replica.cc:282] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:51.528980 17413 raft_consensus.cc:2156] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:51.529004 17413 raft_consensus.cc:2183] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:51.529271 17413 ts_tablet_manager.cc:1457] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:51.531005 17413 ts_tablet_manager.cc:1470] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9 I1121 10:21:51.531047 17413 log.cc:1153] T 6765d5e95b8f45d4a8f0bb9e5d76b9f0 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/6765d5e95b8f45d4a8f0bb9e5d76b9f0 I1121 10:21:51.531410 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 (table rebalance_test_table_9 [id=0cd2841057dc4204a2032bf9c9853863]) successfully deleted I1121 10:21:51.538069 17574 ts_tablet_manager.cc:721] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035) I1121 10:21:51.538117 17574 tablet_copy_client.cc:243] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:43035 I1121 10:21:51.538388 10845 tablet_copy_service.cc:135] P cd693d36f0014dc3a7c3f26196124794: Received BeginTabletCopySession request for tablet 40e235c631e14bf8b4fe83e2953d8c87 from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:50887) I1121 10:21:51.538425 10845 tablet_copy_service.cc:156] P cd693d36f0014dc3a7c3f26196124794: Beginning new tablet copy session on tablet 40e235c631e14bf8b4fe83e2953d8c87 from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:50887: session id = 80d2057cd4814fd3bfe268270d1abdd4-40e235c631e14bf8b4fe83e2953d8c87 I1121 10:21:51.538838 10845 tablet_copy_source_session.cc:204] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:51.539191 17574 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 40e235c631e14bf8b4fe83e2953d8c87. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:51.540052 17574 tablet_copy_client.cc:581] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:51.540133 17574 tablet_copy_client.cc:544] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:51.540729 17574 tablet_copy_client.cc:755] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:51.597477 11080 consensus_queue.cc:208] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:51.598040 11205 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) I1121 10:21:51.598045 12636 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch) W1121 10:21:51.598176 11012 consensus_peers.cc:473] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 -> Peer 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291): Couldn't send request to peer 80d2057cd4814fd3bfe268270d1abdd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8963dd2bc2da4cacb9d5e5329680d7d7. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:51.598248 17612 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:51.598368 17507 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:21:51.599141 16799 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:51.599315 11205 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:51.599315 12636 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:51.600483 10702 catalog_manager.cc:4143] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 6 to 7, NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) added. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } } I1121 10:21:51.745837 17275 tablet_copy_client.cc:420] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:51.746626 17275 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:51.749948 17275 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:51.750182 17275 tablet_bootstrap.cc:439] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:51.750286 17275 ts_tablet_manager.cc:1104] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I1121 10:21:51.750427 17275 raft_consensus.cc:337] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:51.750449 17275 raft_consensus.cc:702] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:51.750494 17275 consensus_queue.cc:231] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:51.750577 17275 ts_tablet_manager.cc:1132] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:51.750874 10845 tablet_copy_service.cc:325] P cd693d36f0014dc3a7c3f26196124794: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-702635e315b940138670eeb5ae6ace1b received from {username='slave'} at 127.5.70.9:50887 I1121 10:21:51.750905 10845 tablet_copy_service.cc:413] P cd693d36f0014dc3a7c3f26196124794: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-702635e315b940138670eeb5ae6ace1b on tablet 702635e315b940138670eeb5ae6ace1b with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:51.842499 17280 tablet_copy_client.cc:420] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:51.843360 17280 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:51.847079 17280 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:51.847249 17280 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:51.847357 17280 ts_tablet_manager.cc:1104] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s I1121 10:21:51.847503 17280 raft_consensus.cc:337] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:51.847527 17280 raft_consensus.cc:702] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:51.847582 17280 consensus_queue.cc:231] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:51.847666 17280 ts_tablet_manager.cc:1132] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:51.848012 10977 tablet_copy_service.cc:325] P af0d02c642ed4eab8d971d9f05d32044: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-4ecac46041da45aba6e82df02354fe89 received from {username='slave'} at 127.5.70.9:35497 I1121 10:21:51.848042 10977 tablet_copy_service.cc:413] P af0d02c642ed4eab8d971d9f05d32044: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-4ecac46041da45aba6e82df02354fe89 on tablet 4ecac46041da45aba6e82df02354fe89 with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:51.908485 17327 tablet_copy_client.cc:420] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:51.909343 17327 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:51.913640 17327 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:51.913796 17327 tablet_bootstrap.cc:439] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:51.913873 17327 ts_tablet_manager.cc:1104] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s I1121 10:21:51.914011 17327 raft_consensus.cc:337] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:51.914029 17327 raft_consensus.cc:702] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:51.914070 17327 consensus_queue.cc:231] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:51.914160 17327 ts_tablet_manager.cc:1132] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:51.914552 11218 tablet_copy_service.cc:325] P 5457a73f9ecb40ee9889513a8412937b: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-df2330a22c8142ada6d056401d0a064d received from {username='slave'} at 127.5.70.9:44295 I1121 10:21:51.914584 11218 tablet_copy_service.cc:413] P 5457a73f9ecb40ee9889513a8412937b: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-df2330a22c8142ada6d056401d0a064d on tablet df2330a22c8142ada6d056401d0a064d with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:52.127288 17327 ts_tablet_manager.cc:721] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: Initiating tablet copy from peer ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217) I1121 10:21:52.127351 17327 tablet_copy_client.cc:243] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.3:40217 I1121 10:21:52.127631 11085 tablet_copy_service.cc:135] P ca8a596d4c4a4b438860e5e0420e3ad6: Received BeginTabletCopySession request for tablet 8963dd2bc2da4cacb9d5e5329680d7d7 from peer 80d2057cd4814fd3bfe268270d1abdd4 ({username='slave'} at 127.5.70.9:41607) I1121 10:21:52.127672 11085 tablet_copy_service.cc:156] P ca8a596d4c4a4b438860e5e0420e3ad6: Beginning new tablet copy session on tablet 8963dd2bc2da4cacb9d5e5329680d7d7 from peer 80d2057cd4814fd3bfe268270d1abdd4 at {username='slave'} at 127.5.70.9:41607: session id = 80d2057cd4814fd3bfe268270d1abdd4-8963dd2bc2da4cacb9d5e5329680d7d7 I1121 10:21:52.128114 11085 tablet_copy_source_session.cc:204] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:52.128542 17327 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8963dd2bc2da4cacb9d5e5329680d7d7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:52.129658 17327 tablet_copy_client.cc:581] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 0 data blocks... I1121 10:21:52.129742 17327 tablet_copy_client.cc:544] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Starting download of 1 WAL segments... I1121 10:21:52.130245 17327 tablet_copy_client.cc:755] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Injecting latency into file download: 1500 I1121 10:21:52.152961 13560 raft_consensus.cc:1123] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:52.228447 13560 raft_consensus.cc:1123] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:52.264470 13560 raft_consensus.cc:1123] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:52.701272 16888 raft_consensus.cc:971] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:52.701628 16888 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:52.702036 13560 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:52.702260 12636 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:52.702407 17745 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:52.702735 17745 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:52.702817 11080 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer af0d02c642ed4eab8d971d9f05d32044: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:52.703065 17745 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:52.703725 17762 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.703863 12636 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.704074 11080 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.705276 10710 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "af0d02c642ed4eab8d971d9f05d32044" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.705281 13560 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.713150 10953 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" mode: GRACEFUL from {username='slave'} at 127.0.0.1:50264 I1121 10:21:52.713193 10953 raft_consensus.cc:581] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:52.726588 17678 raft_consensus.cc:971] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:52.726809 17678 consensus_queue.cc:208] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:52.727357 13560 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:52.727397 11080 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:52.727430 12636 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:52.727628 17788 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:52.727772 17678 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:52.727887 17789 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:52.728972 17788 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.729229 12636 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.729475 11080 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.730381 13560 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.730487 10705 catalog_manager.cc:4143] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.766964 17793 raft_consensus.cc:971] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:52.767230 17793 consensus_queue.cc:208] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:52.767870 10815 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:52.768124 13560 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:52.768152 12636 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:52.768245 17793 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:52.768360 17795 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:52.768476 17796 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:52.769506 17793 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.769666 13560 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.769675 12636 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.770813 10702 catalog_manager.cc:4143] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.771152 10815 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.772294 11205 consensus_queue.cc:208] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:52.772718 12636 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:52.772743 13560 raft_consensus.cc:1181] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Refusing update from remote peer 5457a73f9ecb40ee9889513a8412937b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:52.772920 17800 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:52.773061 17793 consensus_queue.cc:984] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:52.774029 17800 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 5457a73f9ecb40ee9889513a8412937b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.774152 13560 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.774303 12636 raft_consensus.cc:2801] T df2330a22c8142ada6d056401d0a064d P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.775223 10693 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet df2330a22c8142ada6d056401d0a064d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:52.775235 10705 catalog_manager.cc:4143] T df2330a22c8142ada6d056401d0a064d P 80d2057cd4814fd3bfe268270d1abdd4 reported cstate change: config changed from index 8 to 9, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "5457a73f9ecb40ee9889513a8412937b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:52.776191 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet df2330a22c8142ada6d056401d0a064d with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:52.776355 17801 tablet_replica.cc:282] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:52.776443 17801 raft_consensus.cc:2156] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:52.776471 17801 raft_consensus.cc:2183] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:52.776986 17801 ts_tablet_manager.cc:1457] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:52.778975 17801 ts_tablet_manager.cc:1470] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:52.779029 17801 log.cc:1153] T df2330a22c8142ada6d056401d0a064d P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/df2330a22c8142ada6d056401d0a064d I1121 10:21:52.779532 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet df2330a22c8142ada6d056401d0a064d (table rebalance_test_table_4 [id=309f6484f92a48a499fd554b949734f5]) successfully deleted I1121 10:21:52.934906 10815 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" dest_uuid: "cd693d36f0014dc3a7c3f26196124794" mode: GRACEFUL from {username='slave'} at 127.0.0.1:34896 I1121 10:21:52.935163 10815 raft_consensus.cc:581] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:52.943019 10953 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" mode: GRACEFUL from {username='slave'} at 127.0.0.1:50278 I1121 10:21:52.943059 10953 raft_consensus.cc:581] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:52.964907 10815 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" dest_uuid: "cd693d36f0014dc3a7c3f26196124794" mode: GRACEFUL from {username='slave'} at 127.0.0.1:34910 I1121 10:21:52.964956 10815 raft_consensus.cc:581] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:52.973608 10953 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" mode: GRACEFUL from {username='slave'} at 127.0.0.1:50292 I1121 10:21:52.973656 10953 raft_consensus.cc:581] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:53.040978 17574 tablet_copy_client.cc:420] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:53.041815 17574 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:53.046219 17574 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:53.046540 17574 tablet_bootstrap.cc:439] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:53.046674 17574 ts_tablet_manager.cc:1104] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s I1121 10:21:53.046846 17574 raft_consensus.cc:337] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:53.046892 17574 raft_consensus.cc:702] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:53.046947 17574 consensus_queue.cc:231] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:53.047055 17574 ts_tablet_manager.cc:1132] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:53.047523 10845 tablet_copy_service.cc:325] P cd693d36f0014dc3a7c3f26196124794: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-40e235c631e14bf8b4fe83e2953d8c87 received from {username='slave'} at 127.5.70.9:50887 I1121 10:21:53.047572 10845 tablet_copy_service.cc:413] P cd693d36f0014dc3a7c3f26196124794: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-40e235c631e14bf8b4fe83e2953d8c87 on tablet 40e235c631e14bf8b4fe83e2953d8c87 with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:53.058274 13560 raft_consensus.cc:1123] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:53.127972 17678 raft_consensus.cc:900] Instructing follower 80d2057cd4814fd3bfe268270d1abdd4 to start an election I1121 10:21:53.128041 17678 raft_consensus.cc:988] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Signalling peer 80d2057cd4814fd3bfe268270d1abdd4to start an election I1121 10:21:53.128445 13560 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" dest_uuid: "80d2057cd4814fd3bfe268270d1abdd4" from {username='slave'} at 127.5.70.1:37317 I1121 10:21:53.128485 13560 raft_consensus.cc:469] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:21:53.128494 13560 raft_consensus.cc:2905] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:53.129276 13560 raft_consensus.cc:491] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:53.129427 13560 leader_election.cc:296] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [CANDIDATE]: Term 2 election: Requested vote from peers cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471) I1121 10:21:53.129703 10815 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" candidate_uuid: "80d2057cd4814fd3bfe268270d1abdd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "cd693d36f0014dc3a7c3f26196124794" I1121 10:21:53.129705 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" candidate_uuid: "80d2057cd4814fd3bfe268270d1abdd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:53.129735 10815 raft_consensus.cc:2900] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:21:53.129739 11080 raft_consensus.cc:2905] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:53.129765 10815 raft_consensus.cc:702] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd693d36f0014dc3a7c3f26196124794, State: Running, Role: LEADER I1121 10:21:53.129868 10815 consensus_queue.cc:231] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:53.129920 10815 raft_consensus.cc:2905] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:53.130583 11080 raft_consensus.cc:2359] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d2057cd4814fd3bfe268270d1abdd4 in term 2. I1121 10:21:53.130722 10815 raft_consensus.cc:2359] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d2057cd4814fd3bfe268270d1abdd4 in term 2. I1121 10:21:53.130901 13491 leader_election.cc:310] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 80d2057cd4814fd3bfe268270d1abdd4, ca8a596d4c4a4b438860e5e0420e3ad6, cd693d36f0014dc3a7c3f26196124794; no voters: I1121 10:21:53.131023 17412 raft_consensus.cc:2653] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:21:53.131134 17412 raft_consensus.cc:664] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 LEADER]: Becoming Leader. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Running, Role: LEADER I1121 10:21:53.131233 17412 consensus_queue.cc:208] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:53.132135 10702 catalog_manager.cc:4143] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 reported cstate change: term changed from 1 to 2, leader changed from cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) to 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9). New cstate: current_term: 2 leader_uuid: "80d2057cd4814fd3bfe268270d1abdd4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:53.133893 12636 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "702635e315b940138670eeb5ae6ace1b" candidate_uuid: "80d2057cd4814fd3bfe268270d1abdd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" I1121 10:21:53.133935 12636 raft_consensus.cc:2905] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:53.134522 12636 raft_consensus.cc:2359] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80d2057cd4814fd3bfe268270d1abdd4 in term 2. I1121 10:21:53.163552 17756 raft_consensus.cc:900] Instructing follower 97a8ceca326a4f45b15a5b9c4b4857fd to start an election I1121 10:21:53.163643 17763 raft_consensus.cc:988] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Signalling peer 97a8ceca326a4f45b15a5b9c4b4857fdto start an election I1121 10:21:53.164016 12636 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" dest_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" from {username='slave'} at 127.5.70.2:45103 I1121 10:21:53.164049 12636 raft_consensus.cc:469] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:21:53.164058 12636 raft_consensus.cc:2905] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:53.164902 12636 raft_consensus.cc:491] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:53.165016 12636 leader_election.cc:296] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [CANDIDATE]: Term 2 election: Requested vote from peers ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797), 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291) I1121 10:21:53.165382 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:53.165421 11080 raft_consensus.cc:2905] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:53.166065 11080 raft_consensus.cc:2359] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:53.166244 10953 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "af0d02c642ed4eab8d971d9f05d32044" I1121 10:21:53.166285 10953 raft_consensus.cc:2900] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:21:53.166296 10953 raft_consensus.cc:702] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: af0d02c642ed4eab8d971d9f05d32044, State: Running, Role: LEADER I1121 10:21:53.166393 10953 consensus_queue.cc:231] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:53.166442 10953 raft_consensus.cc:2905] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:53.167135 10953 raft_consensus.cc:2359] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:53.167500 12569 leader_election.cc:310] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 97a8ceca326a4f45b15a5b9c4b4857fd, af0d02c642ed4eab8d971d9f05d32044, ca8a596d4c4a4b438860e5e0420e3ad6; no voters: I1121 10:21:53.167670 17279 raft_consensus.cc:2653] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:21:53.167798 17279 raft_consensus.cc:664] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Becoming Leader. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Running, Role: LEADER I1121 10:21:53.167899 17279 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:53.169128 10702 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: term changed from 1 to 2, leader changed from af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) to 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8). New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:53.169309 13560 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "4ecac46041da45aba6e82df02354fe89" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "80d2057cd4814fd3bfe268270d1abdd4" I1121 10:21:53.169342 13560 raft_consensus.cc:2905] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:53.169996 13560 raft_consensus.cc:2359] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:53.548081 17789 raft_consensus.cc:971] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:53.548316 17789 consensus_queue.cc:208] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:53.548844 13560 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:53.548898 10953 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:53.548907 12264 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:53.549100 17916 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:53.549196 17789 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:53.549330 17788 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:53.550246 17789 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.551348 12264 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.551379 10953 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.551524 10702 catalog_manager.cc:4143] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:53.551822 13560 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true } } peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.553954 10815 consensus_queue.cc:208] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:53.554406 12264 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:53.554416 13560 raft_consensus.cc:1181] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Refusing update from remote peer cd693d36f0014dc3a7c3f26196124794: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch) I1121 10:21:53.554754 17789 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:53.554826 17916 consensus_queue.cc:984] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:53.555526 17916 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.555817 12264 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P fed6a935d10042b48cdf1f30824f6c57 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.555980 13560 raft_consensus.cc:2801] T 40e235c631e14bf8b4fe83e2953d8c87 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.556504 10690 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 40e235c631e14bf8b4fe83e2953d8c87 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:53.556946 10705 catalog_manager.cc:4143] T 40e235c631e14bf8b4fe83e2953d8c87 P cd693d36f0014dc3a7c3f26196124794 reported cstate change: config changed from index 8 to 9, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 1 leader_uuid: "cd693d36f0014dc3a7c3f26196124794" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fed6a935d10042b48cdf1f30824f6c57" member_type: VOTER last_known_addr { host: "127.5.70.5" port: 35757 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:53.557803 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 40e235c631e14bf8b4fe83e2953d8c87 with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:53.558100 17921 tablet_replica.cc:282] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:53.558156 17921 raft_consensus.cc:2156] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:21:53.558216 17921 raft_consensus.cc:2183] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044 [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:21:53.558576 17921 ts_tablet_manager.cc:1457] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:53.560398 17921 ts_tablet_manager.cc:1470] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:53.560446 17921 log.cc:1153] T 40e235c631e14bf8b4fe83e2953d8c87 P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/40e235c631e14bf8b4fe83e2953d8c87 I1121 10:21:53.561026 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 40e235c631e14bf8b4fe83e2953d8c87 (table rebalance_test_table_8 [id=759ca891205642f18704c2f78fb1e6c9]) successfully deleted I1121 10:21:53.630453 17327 tablet_copy_client.cc:420] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:53.631356 17327 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap starting. I1121 10:21:53.635867 17327 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:53.648943 17327 tablet_bootstrap.cc:439] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: Bootstrap complete. I1121 10:21:53.649083 17327 ts_tablet_manager.cc:1104] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s I1121 10:21:53.649340 17327 raft_consensus.cc:337] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:53.649359 17327 raft_consensus.cc:702] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80d2057cd4814fd3bfe268270d1abdd4, State: Initialized, Role: LEARNER I1121 10:21:53.649488 17327 consensus_queue.cc:231] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: NON_VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: true } } I1121 10:21:53.649575 17327 ts_tablet_manager.cc:1132] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:53.649787 11085 tablet_copy_service.cc:325] P ca8a596d4c4a4b438860e5e0420e3ad6: Request end of tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-8963dd2bc2da4cacb9d5e5329680d7d7 received from {username='slave'} at 127.5.70.9:41607 I1121 10:21:53.649816 11085 tablet_copy_service.cc:413] P ca8a596d4c4a4b438860e5e0420e3ad6: ending tablet copy session 80d2057cd4814fd3bfe268270d1abdd4-8963dd2bc2da4cacb9d5e5329680d7d7 on tablet 8963dd2bc2da4cacb9d5e5329680d7d7 with peer 80d2057cd4814fd3bfe268270d1abdd4 I1121 10:21:53.657109 11080 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer 80d2057cd4814fd3bfe268270d1abdd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:53.657573 17871 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:53.683615 10815 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Refusing update from remote peer 80d2057cd4814fd3bfe268270d1abdd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:53.684052 17871 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd693d36f0014dc3a7c3f26196124794" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 43035 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:53.686075 12636 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Refusing update from remote peer 80d2057cd4814fd3bfe268270d1abdd4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:53.686329 17961 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:53.705910 11080 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:53.706439 17965 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:53.748778 13560 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:53.749153 17965 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:53.750777 10953 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:53.750993 17965 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0d02c642ed4eab8d971d9f05d32044" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 43797 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:53.751314 12636 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:53.751745 13560 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:53.752050 17965 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:53.752275 11080 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:53.752661 17968 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:53.752876 17874 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.753203 11080 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.754160 10691 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 4ecac46041da45aba6e82df02354fe89 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:53.754180 13560 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:53.754344 10702 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 8 to 10, VOTER af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2) evicted. New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:53.755360 10935 tablet_service.cc:1067] Processing DeleteTablet for tablet 4ecac46041da45aba6e82df02354fe89 with delete_type TABLET_DATA_TOMBSTONED (TS af0d02c642ed4eab8d971d9f05d32044 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48402 I1121 10:21:53.755429 17921 tablet_replica.cc:282] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: stopping tablet replica I1121 10:21:53.755478 17921 raft_consensus.cc:2156] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:53.755497 17921 raft_consensus.cc:2183] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:53.756017 17921 ts_tablet_manager.cc:1457] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:53.758091 17921 ts_tablet_manager.cc:1470] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9 I1121 10:21:53.758131 17921 log.cc:1153] T 4ecac46041da45aba6e82df02354fe89 P af0d02c642ed4eab8d971d9f05d32044: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/wals/4ecac46041da45aba6e82df02354fe89 I1121 10:21:53.758653 10692 catalog_manager.cc:3512] TS af0d02c642ed4eab8d971d9f05d32044 (127.5.70.2:43797): tablet 4ecac46041da45aba6e82df02354fe89 (table rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]) successfully deleted I1121 10:21:54.039181 13560 raft_consensus.cc:1123] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[] I1121 10:21:54.144306 13560 consensus_queue.cc:208] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:54.144937 11080 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer 80d2057cd4814fd3bfe268270d1abdd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:54.144975 12636 raft_consensus.cc:1181] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Refusing update from remote peer 80d2057cd4814fd3bfe268270d1abdd4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:54.145195 17871 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:54.145378 17871 consensus_queue.cc:984] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:54.146414 17871 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:54.146596 11080 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:54.146795 12636 raft_consensus.cc:2801] T 702635e315b940138670eeb5ae6ace1b P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:54.147783 10702 catalog_manager.cc:4143] T 702635e315b940138670eeb5ae6ace1b P 80d2057cd4814fd3bfe268270d1abdd4 reported cstate change: config changed from index 8 to 10, VOTER cd693d36f0014dc3a7c3f26196124794 (127.5.70.1) evicted. New cstate: current_term: 2 leader_uuid: "80d2057cd4814fd3bfe268270d1abdd4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:54.148156 10691 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 702635e315b940138670eeb5ae6ace1b with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:54.148967 10804 tablet_service.cc:1067] Processing DeleteTablet for tablet 702635e315b940138670eeb5ae6ace1b with delete_type TABLET_DATA_TOMBSTONED (TS cd693d36f0014dc3a7c3f26196124794 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33028 I1121 10:21:54.149127 18009 tablet_replica.cc:282] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: stopping tablet replica I1121 10:21:54.149322 18009 raft_consensus.cc:2156] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:54.149335 18009 raft_consensus.cc:2183] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:54.150166 18009 ts_tablet_manager.cc:1457] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:54.152272 18009 ts_tablet_manager.cc:1470] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9 I1121 10:21:54.152317 18009 log.cc:1153] T 702635e315b940138670eeb5ae6ace1b P cd693d36f0014dc3a7c3f26196124794: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/702635e315b940138670eeb5ae6ace1b I1121 10:21:54.152876 10690 catalog_manager.cc:3512] TS cd693d36f0014dc3a7c3f26196124794 (127.5.70.1:43035): tablet 702635e315b940138670eeb5ae6ace1b (table rebalance_test_table_2 [id=40232a9c030148439c3b72bce7b4e378]) successfully deleted I1121 10:21:54.575673 17867 raft_consensus.cc:971] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: attempting to promote NON_VOTER 80d2057cd4814fd3bfe268270d1abdd4 to VOTER I1121 10:21:54.575942 17867 consensus_queue.cc:208] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:54.576417 13560 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 LEARNER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:54.576709 12636 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:54.576786 11202 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Refusing update from remote peer ca8a596d4c4a4b438860e5e0420e3ad6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch) I1121 10:21:54.576958 18033 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:54.577076 17867 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:54.577575 17867 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I1121 10:21:54.579021 18033 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:54.579156 13560 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:54.579177 11202 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:54.580358 12636 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:54.580430 10705 catalog_manager.cc:4143] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 reported cstate change: config changed from index 7 to 8, 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:54.714500 11080 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:54056 I1121 10:21:54.714540 11080 raft_consensus.cc:581] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:54.923810 11080 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" mode: GRACEFUL from {username='slave'} at 127.0.0.1:54060 I1121 10:21:54.923876 11080 raft_consensus.cc:581] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Received request to transfer leadership I1121 10:21:54.986955 18030 raft_consensus.cc:900] Instructing follower 97a8ceca326a4f45b15a5b9c4b4857fd to start an election I1121 10:21:54.987013 18030 raft_consensus.cc:988] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Signalling peer 97a8ceca326a4f45b15a5b9c4b4857fdto start an election I1121 10:21:54.987416 12636 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" dest_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" from {username='slave'} at 127.5.70.3:59439 I1121 10:21:54.987463 12636 raft_consensus.cc:469] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:21:54.987476 12636 raft_consensus.cc:2905] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:54.988215 12636 raft_consensus.cc:491] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:54.988334 12636 leader_election.cc:296] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [CANDIDATE]: Term 2 election: Requested vote from peers 5457a73f9ecb40ee9889513a8412937b (127.5.70.4:37207), ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217), 80d2057cd4814fd3bfe268270d1abdd4 (127.5.70.9:39291) I1121 10:21:54.988628 11202 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "5457a73f9ecb40ee9889513a8412937b" I1121 10:21:54.988641 13560 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "80d2057cd4814fd3bfe268270d1abdd4" I1121 10:21:54.988667 11202 raft_consensus.cc:2905] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:54.988672 13560 raft_consensus.cc:2905] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:54.988742 11080 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8963dd2bc2da4cacb9d5e5329680d7d7" candidate_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" I1121 10:21:54.988768 11080 raft_consensus.cc:2900] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:21:54.988777 11080 raft_consensus.cc:702] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ca8a596d4c4a4b438860e5e0420e3ad6, State: Running, Role: LEADER I1121 10:21:54.988867 11080 consensus_queue.cc:231] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:54.988910 11080 raft_consensus.cc:2905] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 1 FOLLOWER]: Advancing to term 2 I1121 10:21:54.989403 11202 raft_consensus.cc:2359] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:54.989426 13560 raft_consensus.cc:2359] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:54.989580 12568 leader_election.cc:310] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5457a73f9ecb40ee9889513a8412937b, 80d2057cd4814fd3bfe268270d1abdd4, 97a8ceca326a4f45b15a5b9c4b4857fd; no voters: I1121 10:21:54.989653 11080 raft_consensus.cc:2359] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97a8ceca326a4f45b15a5b9c4b4857fd in term 2. I1121 10:21:54.989778 17874 raft_consensus.cc:2653] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:21:54.989934 17874 raft_consensus.cc:664] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Becoming Leader. State: Replica: 97a8ceca326a4f45b15a5b9c4b4857fd, State: Running, Role: LEADER I1121 10:21:54.990034 17874 consensus_queue.cc:208] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:54.990847 10710 catalog_manager.cc:4143] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: term changed from 1 to 2, leader changed from ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) to 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8). New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:21:55.387944 13560 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:55.388461 17874 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:55.404965 11202 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:55.405442 17874 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:55.407150 11080 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch) I1121 10:21:55.407436 18057 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I1121 10:21:55.407665 12636 consensus_queue.cc:208] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } I1121 10:21:55.408113 13560 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:55.408226 11202 raft_consensus.cc:1181] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch) I1121 10:21:55.408434 18057 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:55.408506 17874 consensus_queue.cc:984] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s I1121 10:21:55.409371 18057 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:55.409443 13560 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:55.409456 11202 raft_consensus.cc:2801] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 5457a73f9ecb40ee9889513a8412937b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:55.410377 10691 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 8963dd2bc2da4cacb9d5e5329680d7d7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:55.410439 10702 catalog_manager.cc:4143] T 8963dd2bc2da4cacb9d5e5329680d7d7 P 80d2057cd4814fd3bfe268270d1abdd4 reported cstate change: config changed from index 8 to 10, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "5457a73f9ecb40ee9889513a8412937b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 37207 } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } } I1121 10:21:55.411273 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 8963dd2bc2da4cacb9d5e5329680d7d7 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:55.411487 18074 tablet_replica.cc:282] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:55.411540 18074 raft_consensus.cc:2156] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:55.411564 18074 raft_consensus.cc:2183] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:55.411818 18074 ts_tablet_manager.cc:1457] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:55.413810 18074 ts_tablet_manager.cc:1470] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8 I1121 10:21:55.413856 18074 log.cc:1153] T 8963dd2bc2da4cacb9d5e5329680d7d7 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/8963dd2bc2da4cacb9d5e5329680d7d7 I1121 10:21:55.414341 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 8963dd2bc2da4cacb9d5e5329680d7d7 (table rebalance_test_table_3 [id=96765ac571c94f609bb53a85e8462a9f]) successfully deleted I1121 10:21:55.651551 12637 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:55.652555 13560 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch) I1121 10:21:55.652711 11080 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch) W1121 10:21:55.652873 12568 consensus_peers.cc:473] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd -> Peer 77830a7609a549e0a1c76586d0e301ce (127.5.70.7:40193): Couldn't send request to peer 77830a7609a549e0a1c76586d0e301ce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ecac46041da45aba6e82df02354fe89. This is attempt 1: this message will repeat every 5th retry. I1121 10:21:55.653242 18071 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I1121 10:21:55.653430 18071 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s I1121 10:21:55.655328 18057 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:55.655786 13560 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:55.656138 11080 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } } I1121 10:21:55.658293 10702 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 10 to 11, NON_VOTER 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) added. New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:21:56.172802 18151 ts_tablet_manager.cc:721] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: Initiating tablet copy from peer 97a8ceca326a4f45b15a5b9c4b4857fd (127.5.70.8:41471) I1121 10:21:56.172863 18151 tablet_copy_client.cc:243] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.8:41471 I1121 10:21:56.176020 12657 tablet_copy_service.cc:135] P 97a8ceca326a4f45b15a5b9c4b4857fd: Received BeginTabletCopySession request for tablet 4ecac46041da45aba6e82df02354fe89 from peer 77830a7609a549e0a1c76586d0e301ce ({username='slave'} at 127.5.70.7:60027) I1121 10:21:56.176060 12657 tablet_copy_service.cc:156] P 97a8ceca326a4f45b15a5b9c4b4857fd: Beginning new tablet copy session on tablet 4ecac46041da45aba6e82df02354fe89 from peer 77830a7609a549e0a1c76586d0e301ce at {username='slave'} at 127.5.70.7:60027: session id = 77830a7609a549e0a1c76586d0e301ce-4ecac46041da45aba6e82df02354fe89 I1121 10:21:56.176699 12657 tablet_copy_source_session.cc:204] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:21:56.177287 18151 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 4ecac46041da45aba6e82df02354fe89. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:21:56.178318 18151 tablet_copy_client.cc:581] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 0 data blocks... I1121 10:21:56.178416 18151 tablet_copy_client.cc:544] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Starting download of 1 WAL segments... I1121 10:21:56.179317 18151 tablet_copy_client.cc:755] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Injecting latency into file download: 1500 I1121 10:21:57.679658 18151 tablet_copy_client.cc:420] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:21:57.680704 18151 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap starting. I1121 10:21:57.684645 18151 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:21:57.700757 18151 tablet_bootstrap.cc:439] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: Bootstrap complete. I1121 10:21:57.700935 18151 ts_tablet_manager.cc:1104] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s I1121 10:21:57.701125 18151 raft_consensus.cc:337] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce [term 2 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:57.701155 18151 raft_consensus.cc:702] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 77830a7609a549e0a1c76586d0e301ce, State: Initialized, Role: LEARNER I1121 10:21:57.701227 18151 consensus_queue.cc:231] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: NON_VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: true } } I1121 10:21:57.701344 18151 ts_tablet_manager.cc:1132] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:21:57.701872 12657 tablet_copy_service.cc:325] P 97a8ceca326a4f45b15a5b9c4b4857fd: Request end of tablet copy session 77830a7609a549e0a1c76586d0e301ce-4ecac46041da45aba6e82df02354fe89 received from {username='slave'} at 127.5.70.7:60027 I1121 10:21:57.701925 12657 tablet_copy_service.cc:413] P 97a8ceca326a4f45b15a5b9c4b4857fd: ending tablet copy session 77830a7609a549e0a1c76586d0e301ce-4ecac46041da45aba6e82df02354fe89 on tablet 4ecac46041da45aba6e82df02354fe89 with peer 77830a7609a549e0a1c76586d0e301ce I1121 10:21:58.111799 12512 raft_consensus.cc:1123] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[] I1121 10:21:58.615135 18222 raft_consensus.cc:971] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd: attempting to promote NON_VOTER 77830a7609a549e0a1c76586d0e301ce to VOTER I1121 10:21:58.615384 18222 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:58.615839 13560 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch) I1121 10:21:58.615845 12512 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce [term 2 LEARNER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch) I1121 10:21:58.615845 11080 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch) I1121 10:21:58.616086 18225 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I1121 10:21:58.616191 18222 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I1121 10:21:58.616343 18224 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I1121 10:21:58.617149 18224 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:58.617396 11080 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:58.617589 13560 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:58.618547 10705 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 11 to 12, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:58.618816 12512 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 77830a7609a549e0a1c76586d0e301ce (127.5.70.7) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ca8a596d4c4a4b438860e5e0420e3ad6" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 40217 } attrs { replace: true } } peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:58.620647 12637 consensus_queue.cc:208] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } I1121 10:21:58.620990 13560 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch) I1121 10:21:58.621040 12512 raft_consensus.cc:1181] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce [term 2 FOLLOWER]: Refusing update from remote peer 97a8ceca326a4f45b15a5b9c4b4857fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch) I1121 10:21:58.621187 18224 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s I1121 10:21:58.621325 18225 consensus_queue.cc:984] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s I1121 10:21:58.621953 18225 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:58.622251 13560 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 80d2057cd4814fd3bfe268270d1abdd4 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:58.622299 12512 raft_consensus.cc:2801] T 4ecac46041da45aba6e82df02354fe89 P 77830a7609a549e0a1c76586d0e301ce [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } } } I1121 10:21:58.622874 10691 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 4ecac46041da45aba6e82df02354fe89 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:21:58.622984 10706 catalog_manager.cc:4143] T 4ecac46041da45aba6e82df02354fe89 P 97a8ceca326a4f45b15a5b9c4b4857fd reported cstate change: config changed from index 12 to 13, VOTER ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3) evicted. New cstate: current_term: 2 leader_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "97a8ceca326a4f45b15a5b9c4b4857fd" member_type: VOTER last_known_addr { host: "127.5.70.8" port: 41471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d2057cd4814fd3bfe268270d1abdd4" member_type: VOTER last_known_addr { host: "127.5.70.9" port: 39291 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77830a7609a549e0a1c76586d0e301ce" member_type: VOTER last_known_addr { host: "127.5.70.7" port: 40193 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:21:58.623893 11059 tablet_service.cc:1067] Processing DeleteTablet for tablet 4ecac46041da45aba6e82df02354fe89 with delete_type TABLET_DATA_TOMBSTONED (TS ca8a596d4c4a4b438860e5e0420e3ad6 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:52106 I1121 10:21:58.624048 18234 tablet_replica.cc:282] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6: stopping tablet replica I1121 10:21:58.624114 18234 raft_consensus.cc:2156] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:21:58.624136 18234 raft_consensus.cc:2183] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6 [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:21:58.624466 18234 ts_tablet_manager.cc:1457] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:21:58.625901 18234 ts_tablet_manager.cc:1470] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12 I1121 10:21:58.625936 18234 log.cc:1153] T 4ecac46041da45aba6e82df02354fe89 P ca8a596d4c4a4b438860e5e0420e3ad6: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/wals/4ecac46041da45aba6e82df02354fe89 I1121 10:21:58.626287 10690 catalog_manager.cc:3512] TS ca8a596d4c4a4b438860e5e0420e3ad6 (127.5.70.3:40217): tablet 4ecac46041da45aba6e82df02354fe89 (table rebalance_test_table_0 [id=4b31080619384564a0542383ab3f499b]) successfully deleted Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- c7a94ae451594629a208a04c2a14aa18 | 127.5.70.62:45301 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 2c3332f15c014e68bed03f69d8c6287d | 127.5.70.6:37787 | HEALTHY | 5457a73f9ecb40ee9889513a8412937b | 127.5.70.4:37207 | HEALTHY | 77830a7609a549e0a1c76586d0e301ce | 127.5.70.7:40193 | HEALTHY | 80d2057cd4814fd3bfe268270d1abdd4 | 127.5.70.9:39291 | HEALTHY | 97a8ceca326a4f45b15a5b9c4b4857fd | 127.5.70.8:41471 | HEALTHY | af0d02c642ed4eab8d971d9f05d32044 | 127.5.70.2:43797 | HEALTHY | ca8a596d4c4a4b438860e5e0420e3ad6 | 127.5.70.3:40217 | HEALTHY | cd693d36f0014dc3a7c3f26196124794 | 127.5.70.1:43035 | HEALTHY | fed6a935d10042b48cdf1f30824f6c57 | 127.5.70.5:35757 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 9 Flag | Value | Tags | Tablet Server ---------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 9 server(s) checked local_ip_for_outbound_sockets | 127.5.70.1 | experimental | 127.5.70.1:43035 local_ip_for_outbound_sockets | 127.5.70.2 | experimental | 127.5.70.2:43797 local_ip_for_outbound_sockets | 127.5.70.3 | experimental | 127.5.70.3:40217 local_ip_for_outbound_sockets | 127.5.70.4 | experimental | 127.5.70.4:37207 local_ip_for_outbound_sockets | 127.5.70.5 | experimental | 127.5.70.5:35757 local_ip_for_outbound_sockets | 127.5.70.6 | experimental | 127.5.70.6:37787 local_ip_for_outbound_sockets | 127.5.70.7 | experimental | 127.5.70.7:40193 local_ip_for_outbound_sockets | 127.5.70.8 | experimental | 127.5.70.8:41471 local_ip_for_outbound_sockets | 127.5.70.9 | experimental | 127.5.70.9:39291 never_fsync | true | unsafe,advanced | all 9 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked server_dump_info_format | pb | hidden | all 9 server(s) checked server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.70.1:43035 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.70.2:43797 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.70.3:40217 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.70.4:37207 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.70.5:35757 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.70.6:37787 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.70.7:40193 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.5.70.8:41471 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.TserverAddedDuringRebalancingTest.TserverStarts_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.5.70.9:39291 tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked Version Summary Version | Servers -----------------+-------------------------- 1.12.0-SNAPSHOT | all 10 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ------------------------+----+---------+---------------+---------+------------+------------------+------------- rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 10 First Quartile | 10 Median | 10 Third Quartile | 10 Maximum | 10 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 9 Tables | 10 Tablets | 30 Replicas | 90 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1121 10:21:59.067471 5400 log_verifier.cc:121] Checking tablet 0e5ec5084c6a4d6089c2cb198367d342 I1121 10:21:59.086179 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 0e5ec5084c6a4d6089c2cb198367d342 I1121 10:21:59.086210 5400 log_verifier.cc:121] Checking tablet 0ec67ec228044beda81b53a3e842af49 I1121 10:21:59.105947 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 0ec67ec228044beda81b53a3e842af49 I1121 10:21:59.105993 5400 log_verifier.cc:121] Checking tablet 114dbe156d634fca957db3164f8afabb I1121 10:21:59.132624 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet 114dbe156d634fca957db3164f8afabb I1121 10:21:59.132654 5400 log_verifier.cc:121] Checking tablet 172c695aadee4de6ad0a5a6508b290f6 I1121 10:21:59.161396 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet 172c695aadee4de6ad0a5a6508b290f6 I1121 10:21:59.161427 5400 log_verifier.cc:121] Checking tablet 1fa5ac03774f4944815d262fcdffa2f6 I1121 10:21:59.188881 5400 log_verifier.cc:172] Verified matching terms for 10 ops in tablet 1fa5ac03774f4944815d262fcdffa2f6 I1121 10:21:59.188911 5400 log_verifier.cc:121] Checking tablet 204d260bb58b494091ae91cb57a88ca8 I1121 10:21:59.215548 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 204d260bb58b494091ae91cb57a88ca8 I1121 10:21:59.215580 5400 log_verifier.cc:121] Checking tablet 25b55ff241d8442897ea97523a2cd6fd I1121 10:21:59.243532 5400 log_verifier.cc:172] Verified matching terms for 10 ops in tablet 25b55ff241d8442897ea97523a2cd6fd I1121 10:21:59.243564 5400 log_verifier.cc:121] Checking tablet 3e0bc1c59a1b4e2c9e761a6de8085226 I1121 10:21:59.271265 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet 3e0bc1c59a1b4e2c9e761a6de8085226 I1121 10:21:59.271296 5400 log_verifier.cc:121] Checking tablet 40e235c631e14bf8b4fe83e2953d8c87 I1121 10:21:59.298439 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet 40e235c631e14bf8b4fe83e2953d8c87 I1121 10:21:59.298471 5400 log_verifier.cc:121] Checking tablet 45f0a219c25243fbade25321fcc1067a I1121 10:21:59.324198 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 45f0a219c25243fbade25321fcc1067a I1121 10:21:59.324227 5400 log_verifier.cc:121] Checking tablet 4dfbf2f060fb481fb74acc019b6eaebc I1121 10:21:59.350399 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 4dfbf2f060fb481fb74acc019b6eaebc I1121 10:21:59.350431 5400 log_verifier.cc:121] Checking tablet 4ecac46041da45aba6e82df02354fe89 I1121 10:21:59.377713 5400 log_verifier.cc:172] Verified matching terms for 13 ops in tablet 4ecac46041da45aba6e82df02354fe89 I1121 10:21:59.377753 5400 log_verifier.cc:121] Checking tablet 4ef99b04fd634cd599c435ad3383c4d5 I1121 10:21:59.405171 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 4ef99b04fd634cd599c435ad3383c4d5 I1121 10:21:59.405205 5400 log_verifier.cc:121] Checking tablet 51d8830759cc4842af19cc0d22bb8d65 I1121 10:21:59.432557 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 51d8830759cc4842af19cc0d22bb8d65 I1121 10:21:59.432624 5400 log_verifier.cc:121] Checking tablet 5becd9fee1eb40cf8eebb4ffac223e15 I1121 10:21:59.459508 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 5becd9fee1eb40cf8eebb4ffac223e15 I1121 10:21:59.459545 5400 log_verifier.cc:121] Checking tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 I1121 10:21:59.486848 5400 log_verifier.cc:172] Verified matching terms for 10 ops in tablet 5f7d9348eaaa4477b9dd581c28e2d5e3 I1121 10:21:59.486878 5400 log_verifier.cc:121] Checking tablet 635b29a1f41e418faffe3677886048e5 I1121 10:21:59.513850 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet 635b29a1f41e418faffe3677886048e5 I1121 10:21:59.513880 5400 log_verifier.cc:121] Checking tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 I1121 10:21:59.541923 5400 log_verifier.cc:172] Verified matching terms for 10 ops in tablet 6765d5e95b8f45d4a8f0bb9e5d76b9f0 I1121 10:21:59.541954 5400 log_verifier.cc:121] Checking tablet 69ac58ad89e447a494fc407037d27f8d I1121 10:21:59.568992 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet 69ac58ad89e447a494fc407037d27f8d I1121 10:21:59.569020 5400 log_verifier.cc:121] Checking tablet 69c25581de6e4945a167245a4dcbbfe0 I1121 10:21:59.595194 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet 69c25581de6e4945a167245a4dcbbfe0 I1121 10:21:59.595223 5400 log_verifier.cc:121] Checking tablet 702635e315b940138670eeb5ae6ace1b I1121 10:21:59.622033 5400 log_verifier.cc:172] Verified matching terms for 10 ops in tablet 702635e315b940138670eeb5ae6ace1b I1121 10:21:59.622064 5400 log_verifier.cc:121] Checking tablet 7b34e50304ff46e59622de2e979ac7b8 I1121 10:21:59.650563 5400 log_verifier.cc:172] Verified matching terms for 10 ops in tablet 7b34e50304ff46e59622de2e979ac7b8 I1121 10:21:59.650593 5400 log_verifier.cc:121] Checking tablet 8963dd2bc2da4cacb9d5e5329680d7d7 I1121 10:21:59.678174 5400 log_verifier.cc:172] Verified matching terms for 10 ops in tablet 8963dd2bc2da4cacb9d5e5329680d7d7 I1121 10:21:59.678205 5400 log_verifier.cc:121] Checking tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 I1121 10:21:59.705960 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 8bbbf8a8d2df4f76aa83b5fe842b29a6 I1121 10:21:59.705992 5400 log_verifier.cc:121] Checking tablet a40d6cf45df24788adc85f238c9bcc7e I1121 10:21:59.733860 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet a40d6cf45df24788adc85f238c9bcc7e I1121 10:21:59.733893 5400 log_verifier.cc:121] Checking tablet c7de6e746a8246f89d8b65b59803c3af I1121 10:21:59.762384 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet c7de6e746a8246f89d8b65b59803c3af I1121 10:21:59.762418 5400 log_verifier.cc:121] Checking tablet cc68d4b786cd485285707592d3c25f80 I1121 10:21:59.792448 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet cc68d4b786cd485285707592d3c25f80 I1121 10:21:59.792495 5400 log_verifier.cc:121] Checking tablet cc935fc8b0984388a5db91f6e37824fa I1121 10:21:59.826970 5400 log_verifier.cc:172] Verified matching terms for 6 ops in tablet cc935fc8b0984388a5db91f6e37824fa I1121 10:21:59.827001 5400 log_verifier.cc:121] Checking tablet df2330a22c8142ada6d056401d0a064d I1121 10:21:59.862134 5400 log_verifier.cc:172] Verified matching terms for 9 ops in tablet df2330a22c8142ada6d056401d0a064d I1121 10:21:59.862166 5400 log_verifier.cc:121] Checking tablet fa85115a8bbd45dbb7ce1f18b923022a I1121 10:21:59.897375 5400 log_verifier.cc:172] Verified matching terms for 10 ops in tablet fa85115a8bbd45dbb7ce1f18b923022a I1121 10:21:59.897599 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 10745 I1121 10:21:59.904039 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 10879 I1121 10:21:59.908634 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 11003 I1121 10:21:59.912446 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 11127 I1121 10:21:59.916501 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 12187 I1121 10:21:59.919860 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 12311 I1121 10:21:59.923213 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 12435 I1121 10:21:59.927350 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 12559 I1121 10:21:59.931030 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 13481 I1121 10:21:59.934830 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master with pid 10682 2019-11-21T10:21:59Z chronyd exiting [ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (29062 ms) [----------] 1 test from TserverAddedDuringRebalancingTest (29062 ms total) [----------] 1 test from RebalancerAndSingleReplicaTablets [ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 I1121 10:22:00.018776 5400 test_util.cc:225] Using random seed: -1427231916 I1121 10:22:00.019181 5400 ts_itest-base.cc:112] Starting cluster with: I1121 10:22:00.019197 5400 ts_itest-base.cc:113] -------------- I1121 10:22:00.019201 5400 ts_itest-base.cc:114] 4 tablet servers I1121 10:22:00.019203 5400 ts_itest-base.cc:115] 1 replicas per TS I1121 10:22:00.019207 5400 ts_itest-base.cc:116] -------------- 2019-11-21T10:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2019-11-21T10:22:00Z Disabled control of system clock I1121 10:22:00.026252 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:44101 --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --tserver_unresponsive_timeout_ms=3000 --rpc_reuseport=true --builtin_ntp_servers=127.5.70.20:33077 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:00.072031 18505 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:00.072185 18505 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:00.073118 18505 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:00.073139 18505 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W1121 10:22:00.073144 18505 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:00.073148 18505 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W1121 10:22:00.073151 18505 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I1121 10:22:00.074462 18505 master_main.cc:98] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:33077 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --tserver_unresponsive_timeout_ms=3000 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:44101 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.18505 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:00.076107 18505 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:00.077237 18505 hybrid_clock.cc:183] HybridClock initialized: now 1574331720077213 us; error 37 us; skew 500 ppm I1121 10:22:00.077416 18505 fs_manager.cc:261] Metadata directory not provided I1121 10:22:00.077448 18505 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:00.077529 18505 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:00.077548 18505 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:00.078598 18505 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/instance: uuid: "b0d2cd8e977947beba82033b2b85722f" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.078773 18505 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "b0d2cd8e977947beba82033b2b85722f" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.080236 18505 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:22:00.080582 18505 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:00.080689 18505 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:00.081401 18505 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:00.081436 18505 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal uuid: "b0d2cd8e977947beba82033b2b85722f" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.081470 18505 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:00.143468 18505 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:00.147208 18505 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.62:44101 I1121 10:22:00.147500 18505 webserver.cc:382] Webserver started at http://127.5.70.62:41057/ using document root and password file I1121 10:22:00.147732 18505 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb I1121 10:22:00.148295 18566 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:00.150843 18566 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f: Bootstrap starting. I1121 10:22:00.151099 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master as pid 18505 I1121 10:22:00.151388 18566 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f: No blocks or log segments found. Creating new log. I1121 10:22:00.151450 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.70.1:0 --local_ip_for_outbound_sockets=127.5.70.1 --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:44101 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:33077 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:22:00.151670 18566 log.cc:810] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f: Log is configured to *not* fsync() on all Append() calls I1121 10:22:00.152283 18566 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f: No bootstrap required, opened a new log I1121 10:22:00.153769 18566 raft_consensus.cc:337] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d2cd8e977947beba82033b2b85722f" member_type: VOTER } I1121 10:22:00.153831 18566 raft_consensus.cc:363] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:00.153851 18566 raft_consensus.cc:702] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d2cd8e977947beba82033b2b85722f, State: Initialized, Role: FOLLOWER I1121 10:22:00.154006 18566 consensus_queue.cc:231] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d2cd8e977947beba82033b2b85722f" member_type: VOTER } I1121 10:22:00.154040 18566 raft_consensus.cc:377] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:00.154057 18566 raft_consensus.cc:469] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:00.154083 18566 raft_consensus.cc:2905] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:00.154711 18566 raft_consensus.cc:491] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d2cd8e977947beba82033b2b85722f" member_type: VOTER } I1121 10:22:00.154806 18566 leader_election.cc:310] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0d2cd8e977947beba82033b2b85722f; no voters: I1121 10:22:00.154938 18566 leader_election.cc:296] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:00.155154 18569 raft_consensus.cc:2653] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:00.155318 18569 raft_consensus.cc:664] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [term 1 LEADER]: Becoming Leader. State: Replica: b0d2cd8e977947beba82033b2b85722f, State: Running, Role: LEADER I1121 10:22:00.155407 18569 consensus_queue.cc:208] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d2cd8e977947beba82033b2b85722f" member_type: VOTER } I1121 10:22:00.155650 18569 sys_catalog.cc:340] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0d2cd8e977947beba82033b2b85722f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d2cd8e977947beba82033b2b85722f" member_type: VOTER } } I1121 10:22:00.155679 18569 sys_catalog.cc:343] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [sys.catalog]: This master's current role is: LEADER I1121 10:22:00.155691 18566 sys_catalog.cc:340] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "b0d2cd8e977947beba82033b2b85722f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d2cd8e977947beba82033b2b85722f" member_type: VOTER } } I1121 10:22:00.155707 18566 sys_catalog.cc:343] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [sys.catalog]: This master's current role is: LEADER I1121 10:22:00.155779 18569 sys_catalog.cc:340] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0d2cd8e977947beba82033b2b85722f. Latest consensus state: current_term: 1 leader_uuid: "b0d2cd8e977947beba82033b2b85722f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d2cd8e977947beba82033b2b85722f" member_type: VOTER } } I1121 10:22:00.155791 18569 sys_catalog.cc:343] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [sys.catalog]: This master's current role is: LEADER I1121 10:22:00.155910 18566 sys_catalog.cc:428] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f [sys.catalog]: configured and running, proceeding with master startup. W1121 10:22:00.156905 18574 catalog_manager.cc:1159] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f: acquiring CA information for follower catalog manager: Not found: root CA entry not found W1121 10:22:00.156934 18574 catalog_manager.cc:631] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I1121 10:22:00.156971 18573 catalog_manager.cc:1090] Loading table and tablet metadata into memory... I1121 10:22:00.157107 18573 catalog_manager.cc:1101] Initializing Kudu internal certificate authority... I1121 10:22:00.171501 18573 catalog_manager.cc:985] Generated new certificate authority record I1121 10:22:00.172557 18573 catalog_manager.cc:1110] Loading token signing keys... I1121 10:22:00.178413 18573 catalog_manager.cc:4455] T 00000000000000000000000000000000 P b0d2cd8e977947beba82033b2b85722f: Generated new TSK 0 I1121 10:22:00.178524 18573 catalog_manager.cc:1120] Initializing in-progress tserver states... WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:00.193778 18568 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:00.193945 18568 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:00.194727 18568 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:00.194748 18568 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:00.194769 18568 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.1 I1121 10:22:00.195946 18568 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:33077 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:44101 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.18568 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:00.197856 18568 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:00.205184 18568 hybrid_clock.cc:183] HybridClock initialized: now 1574331720205149 us; error 44 us; skew 500 ppm I1121 10:22:00.205389 18568 fs_manager.cc:261] Metadata directory not provided I1121 10:22:00.205435 18568 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:00.205535 18568 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:00.205559 18568 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:00.206707 18568 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.206907 18568 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.208823 18568 fs_manager.cc:508] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s I1121 10:22:00.209106 18568 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:00.209292 18568 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:00.209990 18568 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:00.210031 18568 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.210083 18568 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:00.245314 18568 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:00.245678 18568 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:00.245699 18568 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:00.252568 18568 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.1:44059 I1121 10:22:00.253051 18568 webserver.cc:382] Webserver started at http://127.5.70.1:39607/ using document root and password file I1121 10:22:00.253384 18568 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb I1121 10:22:00.256114 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 18568 I1121 10:22:00.256398 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.70.2:0 --local_ip_for_outbound_sockets=127.5.70.2 --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:44101 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:33077 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:22:00.262435 18698 heartbeater.cc:322] Connected to a master server at 127.5.70.62:44101 I1121 10:22:00.262547 18698 heartbeater.cc:406] Registering TS with master... I1121 10:22:00.262733 18698 heartbeater.cc:455] Master 127.5.70.62:44101 requested a full tablet report, sending... I1121 10:22:00.263264 18528 ts_manager.cc:192] Registered new tserver with Master: 3fcc96c2cb884d6b9c4f1ffa79564c59 (127.5.70.1:44059) I1121 10:22:00.264597 18528 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.1:57483 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:00.295950 18702 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:00.296119 18702 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:00.296919 18702 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:00.296941 18702 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:00.296958 18702 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.2 I1121 10:22:00.298123 18702 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:33077 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:44101 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.18702 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.2 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:00.299912 18702 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:00.301084 18702 hybrid_clock.cc:183] HybridClock initialized: now 1574331720301057 us; error 37 us; skew 500 ppm I1121 10:22:00.301278 18702 fs_manager.cc:261] Metadata directory not provided I1121 10:22:00.301312 18702 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:00.301393 18702 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:00.301412 18702 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:00.302425 18702 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "28e8d1b188ab48e993b0033564c2d09e" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.302589 18702 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "28e8d1b188ab48e993b0033564c2d09e" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.304034 18702 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:22:00.304296 18702 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:00.304394 18702 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:00.305292 18702 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:00.305333 18702 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal uuid: "28e8d1b188ab48e993b0033564c2d09e" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.305367 18702 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:00.318383 18702 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:00.318740 18702 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:00.318761 18702 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:00.323693 18702 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.2:42157 I1121 10:22:00.324003 18702 webserver.cc:382] Webserver started at http://127.5.70.2:46751/ using document root and password file I1121 10:22:00.324280 18702 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb I1121 10:22:00.328729 18823 heartbeater.cc:322] Connected to a master server at 127.5.70.62:44101 I1121 10:22:00.328786 18823 heartbeater.cc:406] Registering TS with master... I1121 10:22:00.328889 18823 heartbeater.cc:455] Master 127.5.70.62:44101 requested a full tablet report, sending... I1121 10:22:00.329174 18528 ts_manager.cc:192] Registered new tserver with Master: 28e8d1b188ab48e993b0033564c2d09e (127.5.70.2:42157) I1121 10:22:00.329699 18528 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.2:55301 I1121 10:22:00.331300 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 18702 I1121 10:22:00.331593 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.70.3:0 --local_ip_for_outbound_sockets=127.5.70.3 --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:44101 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:33077 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:00.368412 18826 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:00.368585 18826 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:00.369393 18826 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:00.369413 18826 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:00.369426 18826 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.3 I1121 10:22:00.370600 18826 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:33077 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:44101 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.18826 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.3 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:00.372483 18826 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:00.373760 18826 hybrid_clock.cc:183] HybridClock initialized: now 1574331720373669 us; error 98 us; skew 500 ppm I1121 10:22:00.373919 18826 fs_manager.cc:261] Metadata directory not provided I1121 10:22:00.373952 18826 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:00.374059 18826 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:00.374079 18826 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:00.375169 18826 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "e1619ca1896449d2bd8a7f68b8ece273" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.375355 18826 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "e1619ca1896449d2bd8a7f68b8ece273" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.376901 18826 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:22:00.377213 18826 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:00.377348 18826 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:00.378039 18826 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:00.378077 18826 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal uuid: "e1619ca1896449d2bd8a7f68b8ece273" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.378126 18826 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:00.435257 18826 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:00.435686 18826 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:00.435704 18826 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:00.441231 18826 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.3:45069 I1121 10:22:00.441543 18826 webserver.cc:382] Webserver started at http://127.5.70.3:40747/ using document root and password file I1121 10:22:00.441802 18826 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb I1121 10:22:00.445783 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 18826 I1121 10:22:00.446074 18947 heartbeater.cc:322] Connected to a master server at 127.5.70.62:44101 I1121 10:22:00.446126 18947 heartbeater.cc:406] Registering TS with master... I1121 10:22:00.446167 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.70.4:0 --local_ip_for_outbound_sockets=127.5.70.4 --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:44101 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:33077 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:22:00.446244 18947 heartbeater.cc:455] Master 127.5.70.62:44101 requested a full tablet report, sending... I1121 10:22:00.446602 18528 ts_manager.cc:192] Registered new tserver with Master: e1619ca1896449d2bd8a7f68b8ece273 (127.5.70.3:45069) I1121 10:22:00.447145 18528 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.3:43907 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:00.484884 18950 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:00.485034 18950 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:00.485870 18950 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:00.485888 18950 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:00.485903 18950 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.4 I1121 10:22:00.487087 18950 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:33077 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:44101 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.18950 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.4 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:00.488911 18950 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:00.490056 18950 hybrid_clock.cc:183] HybridClock initialized: now 1574331720489970 us; error 89 us; skew 500 ppm I1121 10:22:00.490207 18950 fs_manager.cc:261] Metadata directory not provided I1121 10:22:00.490240 18950 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:00.490337 18950 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:00.490360 18950 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:00.491376 18950 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "d996c37c274e44238c372e99e3aaf130" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.491573 18950 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "d996c37c274e44238c372e99e3aaf130" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.493028 18950 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:22:00.493338 18950 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:00.493470 18950 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:00.494190 18950 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:00.494256 18950 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal uuid: "d996c37c274e44238c372e99e3aaf130" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:00.494313 18950 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:00.523093 18950 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:00.523456 18950 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:00.523474 18950 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:00.529172 18950 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.4:46145 I1121 10:22:00.529510 18950 webserver.cc:382] Webserver started at http://127.5.70.4:33715/ using document root and password file I1121 10:22:00.529767 18950 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb I1121 10:22:00.530391 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 18950 I1121 10:22:00.534449 19071 heartbeater.cc:322] Connected to a master server at 127.5.70.62:44101 I1121 10:22:00.534495 19071 heartbeater.cc:406] Registering TS with master... I1121 10:22:00.534600 19071 heartbeater.cc:455] Master 127.5.70.62:44101 requested a full tablet report, sending... I1121 10:22:00.534904 18526 ts_manager.cc:192] Registered new tserver with Master: d996c37c274e44238c372e99e3aaf130 (127.5.70.4:46145) I1121 10:22:00.535411 18526 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.4:32831 I1121 10:22:00.535998 5400 external_mini_cluster.cc:717] 4 TS(s) registered with all masters I1121 10:22:00.541332 18526 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54394: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I1121 10:22:00.546154 18635 tablet_service.cc:1023] Processing CreateTablet for tablet dc66002863e7400a83880fb6283e77b6 (table=TestTable [id=452457baa9a243bc9140759b4face691]), partition=RANGE (key) PARTITION UNBOUNDED I1121 10:22:00.546371 18635 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet dc66002863e7400a83880fb6283e77b6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:00.548589 19083 tablet_bootstrap.cc:439] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Bootstrap starting. I1121 10:22:00.549094 19083 tablet_bootstrap.cc:590] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No blocks or log segments found. Creating new log. I1121 10:22:00.549335 19083 log.cc:810] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Log is configured to *not* fsync() on all Append() calls I1121 10:22:00.549788 19083 tablet_bootstrap.cc:439] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No bootstrap required, opened a new log I1121 10:22:00.549836 19083 ts_tablet_manager.cc:1104] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:00.550994 19083 raft_consensus.cc:337] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:00.551043 19083 raft_consensus.cc:363] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:00.551055 19083 raft_consensus.cc:702] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Initialized, Role: FOLLOWER I1121 10:22:00.551164 19083 consensus_queue.cc:231] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:00.551189 19083 raft_consensus.cc:377] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:00.551200 19083 raft_consensus.cc:469] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:00.551219 19083 raft_consensus.cc:2905] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:00.551795 19083 raft_consensus.cc:491] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:00.551882 19083 leader_election.cc:310] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fcc96c2cb884d6b9c4f1ffa79564c59; no voters: I1121 10:22:00.552012 19083 leader_election.cc:296] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:00.552071 19085 raft_consensus.cc:2653] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:00.552173 19085 raft_consensus.cc:664] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 LEADER]: Becoming Leader. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Running, Role: LEADER I1121 10:22:00.552206 18698 heartbeater.cc:447] Master 127.5.70.62:44101 was elected leader, sending a full tablet report... I1121 10:22:00.552250 19085 consensus_queue.cc:208] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:00.552206 19083 ts_tablet_manager.cc:1132] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s I1121 10:22:00.553017 18526 catalog_manager.cc:4143] T dc66002863e7400a83880fb6283e77b6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 reported cstate change: term changed from 0 to 1, leader changed from to 3fcc96c2cb884d6b9c4f1ffa79564c59 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } health_report { overall_health: HEALTHY } } } I1121 10:22:00.556466 5400 external_mini_cluster.cc:717] 4 TS(s) registered with all masters I1121 10:22:00.556900 5400 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3fcc96c2cb884d6b9c4f1ffa79564c59 to finish bootstrapping I1121 10:22:00.559424 5400 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 28e8d1b188ab48e993b0033564c2d09e to finish bootstrapping I1121 10:22:00.562085 5400 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e1619ca1896449d2bd8a7f68b8ece273 to finish bootstrapping I1121 10:22:00.564805 5400 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d996c37c274e44238c372e99e3aaf130 to finish bootstrapping I1121 10:22:00.567560 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 18826 I1121 10:22:00.570694 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 18950 I1121 10:22:01.330404 18823 heartbeater.cc:447] Master 127.5.70.62:44101 was elected leader, sending a full tablet report... I1121 10:22:04.324453 18526 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54394: name: "rebalance_test_table_0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:22:04.326402 18635 tablet_service.cc:1023] Processing CreateTablet for tablet 71102c6fcd874338bb12a326de2f66eb (table=rebalance_test_table_0 [id=971ac1ce38b848c984644e015410c59d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.326537 18635 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 71102c6fcd874338bb12a326de2f66eb. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.327628 19099 tablet_bootstrap.cc:439] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59: Bootstrap starting. I1121 10:22:04.327960 19099 tablet_bootstrap.cc:590] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59: No blocks or log segments found. Creating new log. I1121 10:22:04.328600 19099 tablet_bootstrap.cc:439] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59: No bootstrap required, opened a new log I1121 10:22:04.328649 19099 ts_tablet_manager.cc:1104] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.328853 19099 raft_consensus.cc:337] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.328881 19099 raft_consensus.cc:363] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.328888 19099 raft_consensus.cc:702] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Initialized, Role: FOLLOWER I1121 10:22:04.328961 19099 consensus_queue.cc:231] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.328976 19099 raft_consensus.cc:377] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.328985 19099 raft_consensus.cc:469] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.328994 19099 raft_consensus.cc:2905] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.329257 18760 tablet_service.cc:1023] Processing CreateTablet for tablet 978a93ad52404bba88734538acfacc4c (table=rebalance_test_table_0 [id=971ac1ce38b848c984644e015410c59d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.329259 18759 tablet_service.cc:1023] Processing CreateTablet for tablet 77515e9cab7d432c8a48ca4751056ebe (table=rebalance_test_table_0 [id=971ac1ce38b848c984644e015410c59d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.329471 18760 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 978a93ad52404bba88734538acfacc4c. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.329591 18759 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 77515e9cab7d432c8a48ca4751056ebe. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.329671 19099 raft_consensus.cc:491] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.329725 19099 leader_election.cc:310] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fcc96c2cb884d6b9c4f1ffa79564c59; no voters: I1121 10:22:04.329799 19099 leader_election.cc:296] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.330155 19099 ts_tablet_manager.cc:1132] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:04.331784 19101 raft_consensus.cc:2653] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.331988 19103 tablet_bootstrap.cc:439] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e: Bootstrap starting. I1121 10:22:04.332077 19101 raft_consensus.cc:664] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 LEADER]: Becoming Leader. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Running, Role: LEADER I1121 10:22:04.332154 19101 consensus_queue.cc:208] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.332676 18526 catalog_manager.cc:4143] T 71102c6fcd874338bb12a326de2f66eb P 3fcc96c2cb884d6b9c4f1ffa79564c59 reported cstate change: term changed from 0 to 1, leader changed from to 3fcc96c2cb884d6b9c4f1ffa79564c59 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.332799 19103 tablet_bootstrap.cc:590] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e: No blocks or log segments found. Creating new log. I1121 10:22:04.333073 19103 log.cc:810] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e: Log is configured to *not* fsync() on all Append() calls I1121 10:22:04.333676 19103 tablet_bootstrap.cc:439] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e: No bootstrap required, opened a new log I1121 10:22:04.333729 19103 ts_tablet_manager.cc:1104] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:22:04.334965 19103 raft_consensus.cc:337] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.335021 19103 raft_consensus.cc:363] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.335041 19103 raft_consensus.cc:702] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Initialized, Role: FOLLOWER I1121 10:22:04.335155 19103 consensus_queue.cc:231] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.335176 19103 raft_consensus.cc:377] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.335186 19103 raft_consensus.cc:469] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.335206 19103 raft_consensus.cc:2905] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.335713 19103 raft_consensus.cc:491] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.335793 19103 leader_election.cc:310] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28e8d1b188ab48e993b0033564c2d09e; no voters: I1121 10:22:04.335911 19103 leader_election.cc:296] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.335968 19107 raft_consensus.cc:2653] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.336057 19103 ts_tablet_manager.cc:1132] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s I1121 10:22:04.336109 19103 tablet_bootstrap.cc:439] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e: Bootstrap starting. I1121 10:22:04.336110 19107 raft_consensus.cc:664] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [term 1 LEADER]: Becoming Leader. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Running, Role: LEADER I1121 10:22:04.336210 19107 consensus_queue.cc:208] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.336431 19103 tablet_bootstrap.cc:590] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e: No blocks or log segments found. Creating new log. I1121 10:22:04.336863 18526 catalog_manager.cc:4143] T 77515e9cab7d432c8a48ca4751056ebe P 28e8d1b188ab48e993b0033564c2d09e reported cstate change: term changed from 0 to 1, leader changed from to 28e8d1b188ab48e993b0033564c2d09e (127.5.70.2). New cstate: current_term: 1 leader_uuid: "28e8d1b188ab48e993b0033564c2d09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.337287 19103 tablet_bootstrap.cc:439] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e: No bootstrap required, opened a new log I1121 10:22:04.337363 19103 ts_tablet_manager.cc:1104] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.337561 19103 raft_consensus.cc:337] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.337596 19103 raft_consensus.cc:363] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.337635 19103 raft_consensus.cc:702] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Initialized, Role: FOLLOWER I1121 10:22:04.337723 19103 consensus_queue.cc:231] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.337744 19103 raft_consensus.cc:377] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.337752 19103 raft_consensus.cc:469] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.337760 19103 raft_consensus.cc:2905] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.338169 19103 raft_consensus.cc:491] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.338199 19103 leader_election.cc:310] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28e8d1b188ab48e993b0033564c2d09e; no voters: I1121 10:22:04.338223 19103 leader_election.cc:296] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.338274 19103 ts_tablet_manager.cc:1132] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.338318 19109 raft_consensus.cc:2653] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.338351 19109 raft_consensus.cc:664] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [term 1 LEADER]: Becoming Leader. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Running, Role: LEADER I1121 10:22:04.338407 19109 consensus_queue.cc:208] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.338852 18526 catalog_manager.cc:4143] T 978a93ad52404bba88734538acfacc4c P 28e8d1b188ab48e993b0033564c2d09e reported cstate change: term changed from 0 to 1, leader changed from to 28e8d1b188ab48e993b0033564c2d09e (127.5.70.2). New cstate: current_term: 1 leader_uuid: "28e8d1b188ab48e993b0033564c2d09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.421586 18526 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54394: name: "rebalance_test_table_1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:22:04.423029 18759 tablet_service.cc:1023] Processing CreateTablet for tablet b4e2be480c6f45d8827b04e898bfe164 (table=rebalance_test_table_1 [id=f7456563768546279dc972d86968f1c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.423055 18635 tablet_service.cc:1023] Processing CreateTablet for tablet 73e224b39f574bfeab6ee341f1a161c6 (table=rebalance_test_table_1 [id=f7456563768546279dc972d86968f1c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.423076 18634 tablet_service.cc:1023] Processing CreateTablet for tablet b50bde701f684114a8a6694a0e7eccc5 (table=rebalance_test_table_1 [id=f7456563768546279dc972d86968f1c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.423164 18759 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b4e2be480c6f45d8827b04e898bfe164. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.423167 18635 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 73e224b39f574bfeab6ee341f1a161c6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.423247 18634 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b50bde701f684114a8a6694a0e7eccc5. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.424129 19099 tablet_bootstrap.cc:439] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Bootstrap starting. I1121 10:22:04.424430 19103 tablet_bootstrap.cc:439] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e: Bootstrap starting. I1121 10:22:04.424482 19099 tablet_bootstrap.cc:590] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No blocks or log segments found. Creating new log. I1121 10:22:04.424860 19103 tablet_bootstrap.cc:590] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e: No blocks or log segments found. Creating new log. I1121 10:22:04.425115 19099 tablet_bootstrap.cc:439] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No bootstrap required, opened a new log I1121 10:22:04.425155 19099 ts_tablet_manager.cc:1104] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.425290 19099 raft_consensus.cc:337] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.425312 19099 raft_consensus.cc:363] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.425319 19099 raft_consensus.cc:702] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Initialized, Role: FOLLOWER I1121 10:22:04.425413 19099 consensus_queue.cc:231] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.425436 19099 raft_consensus.cc:377] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.425443 19099 raft_consensus.cc:469] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.425448 19103 tablet_bootstrap.cc:439] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e: No bootstrap required, opened a new log I1121 10:22:04.425452 19099 raft_consensus.cc:2905] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.425474 19103 ts_tablet_manager.cc:1104] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:04.425608 19103 raft_consensus.cc:337] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.425626 19103 raft_consensus.cc:363] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.425631 19103 raft_consensus.cc:702] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Initialized, Role: FOLLOWER I1121 10:22:04.425673 19103 consensus_queue.cc:231] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.425745 19103 raft_consensus.cc:377] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.425760 19103 raft_consensus.cc:469] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.425776 19103 raft_consensus.cc:2905] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.425979 19099 raft_consensus.cc:491] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.426010 19099 leader_election.cc:310] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fcc96c2cb884d6b9c4f1ffa79564c59; no voters: I1121 10:22:04.426039 19099 leader_election.cc:296] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.426137 19102 raft_consensus.cc:2653] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.426172 19102 raft_consensus.cc:664] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 LEADER]: Becoming Leader. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Running, Role: LEADER I1121 10:22:04.426182 19099 ts_tablet_manager.cc:1132] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.426231 19102 consensus_queue.cc:208] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.426244 19103 raft_consensus.cc:491] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.426239 19099 tablet_bootstrap.cc:439] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Bootstrap starting. I1121 10:22:04.426267 19103 leader_election.cc:310] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28e8d1b188ab48e993b0033564c2d09e; no voters: I1121 10:22:04.426314 19107 raft_consensus.cc:2653] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.426342 19107 raft_consensus.cc:664] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [term 1 LEADER]: Becoming Leader. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Running, Role: LEADER I1121 10:22:04.426384 19107 consensus_queue.cc:208] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.426548 19103 leader_election.cc:296] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.426590 19099 tablet_bootstrap.cc:590] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No blocks or log segments found. Creating new log. I1121 10:22:04.426798 18526 catalog_manager.cc:4143] T 73e224b39f574bfeab6ee341f1a161c6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 reported cstate change: term changed from 0 to 1, leader changed from to 3fcc96c2cb884d6b9c4f1ffa79564c59 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.426968 19103 ts_tablet_manager.cc:1132] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:04.427244 18528 catalog_manager.cc:4143] T b4e2be480c6f45d8827b04e898bfe164 P 28e8d1b188ab48e993b0033564c2d09e reported cstate change: term changed from 0 to 1, leader changed from to 28e8d1b188ab48e993b0033564c2d09e (127.5.70.2). New cstate: current_term: 1 leader_uuid: "28e8d1b188ab48e993b0033564c2d09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.427291 19099 tablet_bootstrap.cc:439] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No bootstrap required, opened a new log I1121 10:22:04.427328 19099 ts_tablet_manager.cc:1104] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:22:04.427475 19099 raft_consensus.cc:337] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.427500 19099 raft_consensus.cc:363] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.427508 19099 raft_consensus.cc:702] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Initialized, Role: FOLLOWER I1121 10:22:04.427578 19099 consensus_queue.cc:231] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.427600 19099 raft_consensus.cc:377] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.427608 19099 raft_consensus.cc:469] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.427620 19099 raft_consensus.cc:2905] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.428141 19099 raft_consensus.cc:491] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.428180 19099 leader_election.cc:310] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fcc96c2cb884d6b9c4f1ffa79564c59; no voters: I1121 10:22:04.428211 19099 leader_election.cc:296] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.428265 19102 raft_consensus.cc:2653] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.428277 19099 ts_tablet_manager.cc:1132] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:22:04.428370 19102 raft_consensus.cc:664] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 LEADER]: Becoming Leader. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Running, Role: LEADER I1121 10:22:04.428416 19102 consensus_queue.cc:208] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.428768 18528 catalog_manager.cc:4143] T b50bde701f684114a8a6694a0e7eccc5 P 3fcc96c2cb884d6b9c4f1ffa79564c59 reported cstate change: term changed from 0 to 1, leader changed from to 3fcc96c2cb884d6b9c4f1ffa79564c59 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.512310 18526 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54394: name: "rebalance_test_table_2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:22:04.513646 18634 tablet_service.cc:1023] Processing CreateTablet for tablet 851f425c2daf42d5a25e4894f2062ab6 (table=rebalance_test_table_2 [id=9f717e3c7ec8493ba8008fd04e84160a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.513655 18759 tablet_service.cc:1023] Processing CreateTablet for tablet e6b46769540548608fc99e63af52cb12 (table=rebalance_test_table_2 [id=9f717e3c7ec8493ba8008fd04e84160a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.513720 18760 tablet_service.cc:1023] Processing CreateTablet for tablet 078d24c361bc4a668606a29cea4d9167 (table=rebalance_test_table_2 [id=9f717e3c7ec8493ba8008fd04e84160a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.513772 18759 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet e6b46769540548608fc99e63af52cb12. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.513772 18634 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 851f425c2daf42d5a25e4894f2062ab6. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.513842 18760 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 078d24c361bc4a668606a29cea4d9167. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.514837 19103 tablet_bootstrap.cc:439] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e: Bootstrap starting. I1121 10:22:04.515107 19099 tablet_bootstrap.cc:439] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Bootstrap starting. I1121 10:22:04.515205 19103 tablet_bootstrap.cc:590] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e: No blocks or log segments found. Creating new log. I1121 10:22:04.515527 19099 tablet_bootstrap.cc:590] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No blocks or log segments found. Creating new log. I1121 10:22:04.515897 19103 tablet_bootstrap.cc:439] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e: No bootstrap required, opened a new log I1121 10:22:04.515952 19103 ts_tablet_manager.cc:1104] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:04.516093 19103 raft_consensus.cc:337] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.516096 19099 tablet_bootstrap.cc:439] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No bootstrap required, opened a new log I1121 10:22:04.516121 19103 raft_consensus.cc:363] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.516130 19103 raft_consensus.cc:702] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Initialized, Role: FOLLOWER I1121 10:22:04.516136 19099 ts_tablet_manager.cc:1104] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:22:04.516193 19103 consensus_queue.cc:231] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.516207 19103 raft_consensus.cc:377] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.516227 19103 raft_consensus.cc:469] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.516237 19103 raft_consensus.cc:2905] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.516281 19099 raft_consensus.cc:337] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.516371 19099 raft_consensus.cc:363] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.516392 19099 raft_consensus.cc:702] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Initialized, Role: FOLLOWER I1121 10:22:04.516466 19099 consensus_queue.cc:231] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.516489 19099 raft_consensus.cc:377] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.516496 19099 raft_consensus.cc:469] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.516503 19099 raft_consensus.cc:2905] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.516968 19103 raft_consensus.cc:491] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.517006 19103 leader_election.cc:310] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28e8d1b188ab48e993b0033564c2d09e; no voters: I1121 10:22:04.517066 19107 raft_consensus.cc:2653] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.517113 19107 raft_consensus.cc:664] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [term 1 LEADER]: Becoming Leader. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Running, Role: LEADER I1121 10:22:04.517159 19107 consensus_queue.cc:208] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.517204 19099 raft_consensus.cc:491] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.517228 19099 leader_election.cc:310] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fcc96c2cb884d6b9c4f1ffa79564c59; no voters: I1121 10:22:04.517277 19099 leader_election.cc:296] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.517318 19102 raft_consensus.cc:2653] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.517350 19102 raft_consensus.cc:664] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 LEADER]: Becoming Leader. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Running, Role: LEADER I1121 10:22:04.517390 19103 leader_election.cc:296] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.517405 19102 consensus_queue.cc:208] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.517477 19103 ts_tablet_manager.cc:1132] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I1121 10:22:04.517508 19103 tablet_bootstrap.cc:439] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e: Bootstrap starting. I1121 10:22:04.517607 18526 catalog_manager.cc:4143] T e6b46769540548608fc99e63af52cb12 P 28e8d1b188ab48e993b0033564c2d09e reported cstate change: term changed from 0 to 1, leader changed from to 28e8d1b188ab48e993b0033564c2d09e (127.5.70.2). New cstate: current_term: 1 leader_uuid: "28e8d1b188ab48e993b0033564c2d09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.517804 19103 tablet_bootstrap.cc:590] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e: No blocks or log segments found. Creating new log. I1121 10:22:04.518033 18528 catalog_manager.cc:4143] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59 reported cstate change: term changed from 0 to 1, leader changed from to 3fcc96c2cb884d6b9c4f1ffa79564c59 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.518399 19103 tablet_bootstrap.cc:439] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e: No bootstrap required, opened a new log I1121 10:22:04.518487 19103 ts_tablet_manager.cc:1104] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.518637 19099 ts_tablet_manager.cc:1132] T 851f425c2daf42d5a25e4894f2062ab6 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I1121 10:22:04.518656 19103 raft_consensus.cc:337] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.518671 19103 raft_consensus.cc:363] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.518678 19103 raft_consensus.cc:702] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Initialized, Role: FOLLOWER I1121 10:22:04.518728 19103 consensus_queue.cc:231] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.518748 19103 raft_consensus.cc:377] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.518754 19103 raft_consensus.cc:469] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.518761 19103 raft_consensus.cc:2905] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.519495 19103 raft_consensus.cc:491] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.519564 19103 leader_election.cc:310] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28e8d1b188ab48e993b0033564c2d09e; no voters: I1121 10:22:04.519604 19103 leader_election.cc:296] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.519620 19107 raft_consensus.cc:2653] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.519655 19107 raft_consensus.cc:664] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [term 1 LEADER]: Becoming Leader. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Running, Role: LEADER I1121 10:22:04.519672 19103 ts_tablet_manager.cc:1132] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.519767 19107 consensus_queue.cc:208] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.520220 18526 catalog_manager.cc:4143] T 078d24c361bc4a668606a29cea4d9167 P 28e8d1b188ab48e993b0033564c2d09e reported cstate change: term changed from 0 to 1, leader changed from to 28e8d1b188ab48e993b0033564c2d09e (127.5.70.2). New cstate: current_term: 1 leader_uuid: "28e8d1b188ab48e993b0033564c2d09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.601991 18526 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54394: name: "rebalance_test_table_3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I1121 10:22:04.603719 18634 tablet_service.cc:1023] Processing CreateTablet for tablet 94fd9d56e2e84494b545ff7dfeeb2472 (table=rebalance_test_table_3 [id=a544b8c298bf4154a9328353a5117294]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.603737 18635 tablet_service.cc:1023] Processing CreateTablet for tablet bfc84ced32954427af051f3b0119983b (table=rebalance_test_table_3 [id=a544b8c298bf4154a9328353a5117294]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.603791 18760 tablet_service.cc:1023] Processing CreateTablet for tablet 6c2dbaff378a4bf1b768e12d054e5992 (table=rebalance_test_table_3 [id=a544b8c298bf4154a9328353a5117294]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:04.603871 18634 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 94fd9d56e2e84494b545ff7dfeeb2472. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.603883 18760 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 6c2dbaff378a4bf1b768e12d054e5992. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.603971 18635 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet bfc84ced32954427af051f3b0119983b. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:04.604943 19099 tablet_bootstrap.cc:439] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Bootstrap starting. I1121 10:22:04.605144 19103 tablet_bootstrap.cc:439] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e: Bootstrap starting. I1121 10:22:04.605491 19099 tablet_bootstrap.cc:590] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No blocks or log segments found. Creating new log. I1121 10:22:04.605491 19103 tablet_bootstrap.cc:590] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e: No blocks or log segments found. Creating new log. I1121 10:22:04.606060 19103 tablet_bootstrap.cc:439] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e: No bootstrap required, opened a new log I1121 10:22:04.606074 19099 tablet_bootstrap.cc:439] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59: No bootstrap required, opened a new log I1121 10:22:04.606092 19103 ts_tablet_manager.cc:1104] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:22:04.606101 19099 ts_tablet_manager.cc:1104] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:04.606215 19103 raft_consensus.cc:337] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.606237 19103 raft_consensus.cc:363] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.606253 19103 raft_consensus.cc:702] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Initialized, Role: FOLLOWER I1121 10:22:04.606258 19099 raft_consensus.cc:337] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.606273 19099 raft_consensus.cc:363] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.606281 19099 raft_consensus.cc:702] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Initialized, Role: FOLLOWER I1121 10:22:04.606323 19103 consensus_queue.cc:231] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.606371 19103 raft_consensus.cc:377] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.606375 19099 consensus_queue.cc:231] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.606389 19103 raft_consensus.cc:469] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.606390 19099 raft_consensus.cc:377] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.606397 19103 raft_consensus.cc:2905] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.606397 19099 raft_consensus.cc:469] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.606406 19099 raft_consensus.cc:2905] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.606837 19103 raft_consensus.cc:491] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.606848 19099 raft_consensus.cc:491] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.606865 19103 leader_election.cc:310] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28e8d1b188ab48e993b0033564c2d09e; no voters: I1121 10:22:04.606868 19099 leader_election.cc:310] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fcc96c2cb884d6b9c4f1ffa79564c59; no voters: I1121 10:22:04.606889 19103 leader_election.cc:296] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.606927 19133 raft_consensus.cc:2653] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.606937 19109 raft_consensus.cc:2653] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.606964 19109 raft_consensus.cc:664] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [term 1 LEADER]: Becoming Leader. State: Replica: 28e8d1b188ab48e993b0033564c2d09e, State: Running, Role: LEADER I1121 10:22:04.606993 19133 raft_consensus.cc:664] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 LEADER]: Becoming Leader. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Running, Role: LEADER I1121 10:22:04.607014 19103 ts_tablet_manager.cc:1132] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.607014 19109 consensus_queue.cc:208] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } } I1121 10:22:04.607071 19133 consensus_queue.cc:208] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.607431 18526 catalog_manager.cc:4143] T 6c2dbaff378a4bf1b768e12d054e5992 P 28e8d1b188ab48e993b0033564c2d09e reported cstate change: term changed from 0 to 1, leader changed from to 28e8d1b188ab48e993b0033564c2d09e (127.5.70.2). New cstate: current_term: 1 leader_uuid: "28e8d1b188ab48e993b0033564c2d09e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28e8d1b188ab48e993b0033564c2d09e" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 42157 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.607468 18528 catalog_manager.cc:4143] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 reported cstate change: term changed from 0 to 1, leader changed from to 3fcc96c2cb884d6b9c4f1ffa79564c59 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.607677 19099 leader_election.cc:296] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.607806 19099 ts_tablet_manager.cc:1132] T 94fd9d56e2e84494b545ff7dfeeb2472 P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I1121 10:22:04.607851 19099 tablet_bootstrap.cc:439] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59: Bootstrap starting. I1121 10:22:04.608178 19099 tablet_bootstrap.cc:590] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59: No blocks or log segments found. Creating new log. I1121 10:22:04.608793 19099 tablet_bootstrap.cc:439] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59: No bootstrap required, opened a new log I1121 10:22:04.608875 19099 ts_tablet_manager.cc:1104] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s I1121 10:22:04.609027 19099 raft_consensus.cc:337] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.609047 19099 raft_consensus.cc:363] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:04.609052 19099 raft_consensus.cc:702] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Initialized, Role: FOLLOWER I1121 10:22:04.609112 19099 consensus_queue.cc:231] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.609133 19099 raft_consensus.cc:377] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:04.609139 19099 raft_consensus.cc:469] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:04.609153 19099 raft_consensus.cc:2905] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:04.609581 19099 raft_consensus.cc:491] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.609611 19099 leader_election.cc:310] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fcc96c2cb884d6b9c4f1ffa79564c59; no voters: I1121 10:22:04.609629 19099 leader_election.cc:296] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:04.609705 19102 raft_consensus.cc:2653] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:04.609730 19099 ts_tablet_manager.cc:1132] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:22:04.609740 19102 raft_consensus.cc:664] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [term 1 LEADER]: Becoming Leader. State: Replica: 3fcc96c2cb884d6b9c4f1ffa79564c59, State: Running, Role: LEADER I1121 10:22:04.609782 19102 consensus_queue.cc:208] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } } I1121 10:22:04.610195 18528 catalog_manager.cc:4143] T bfc84ced32954427af051f3b0119983b P 3fcc96c2cb884d6b9c4f1ffa79564c59 reported cstate change: term changed from 0 to 1, leader changed from to 3fcc96c2cb884d6b9c4f1ffa79564c59 (127.5.70.1). New cstate: current_term: 1 leader_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fcc96c2cb884d6b9c4f1ffa79564c59" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 44059 } health_report { overall_health: HEALTHY } } } I1121 10:22:04.695101 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.70.3:45069 --local_ip_for_outbound_sockets=127.5.70.3 --webserver_port=40747 --webserver_interface=127.5.70.3 --tserver_master_addrs=127.5.70.62:44101 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:33077 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:04.731575 19186 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:04.731734 19186 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:04.732470 19186 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:04.732487 19186 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:04.732501 19186 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.3 I1121 10:22:04.733639 19186 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:33077 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.3:45069 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.70.3 --webserver_port=40747 --tserver_master_addrs=127.5.70.62:44101 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.19186 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.3 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:04.735376 19186 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:04.736538 19186 hybrid_clock.cc:183] HybridClock initialized: now 1574331724736464 us; error 57 us; skew 500 ppm I1121 10:22:04.736723 19186 fs_manager.cc:261] Metadata directory not provided I1121 10:22:04.736788 19186 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:04.738044 19186 fs_manager.cc:404] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.738219 19186 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:04.738350 19186 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:04.739032 19186 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:04.739434 19186 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal uuid: "e1619ca1896449d2bd8a7f68b8ece273" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:04.739490 19186 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:04.767410 19186 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:04.767807 19186 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:04.767827 19186 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:04.773778 19186 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.3:45069 I1121 10:22:04.774211 19186 webserver.cc:382] Webserver started at http://127.5.70.3:40747/ using document root and password file I1121 10:22:04.774507 19186 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb I1121 10:22:04.779104 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 19186 I1121 10:22:04.779315 19307 heartbeater.cc:322] Connected to a master server at 127.5.70.62:44101 I1121 10:22:04.779361 19307 heartbeater.cc:406] Registering TS with master... I1121 10:22:04.779448 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.70.4:46145 --local_ip_for_outbound_sockets=127.5.70.4 --webserver_port=33715 --webserver_interface=127.5.70.4 --tserver_master_addrs=127.5.70.62:44101 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:33077 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:22:04.779464 19307 heartbeater.cc:455] Master 127.5.70.62:44101 requested a full tablet report, sending... I1121 10:22:04.779820 18528 ts_manager.cc:192] Re-registered known tserver with Master: e1619ca1896449d2bd8a7f68b8ece273 (127.5.70.3:45069) I1121 10:22:04.780292 18528 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.3:57585 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:04.817699 19310 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:04.817857 19310 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:04.818608 19310 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:04.818624 19310 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:04.818637 19310 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.4 I1121 10:22:04.819846 19310 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:33077 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.4:46145 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.70.4 --webserver_port=33715 --tserver_master_addrs=127.5.70.62:44101 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.19310 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.4 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:04.821635 19310 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:04.822796 19310 hybrid_clock.cc:183] HybridClock initialized: now 1574331724822770 us; error 27 us; skew 500 ppm I1121 10:22:04.822933 19310 fs_manager.cc:261] Metadata directory not provided I1121 10:22:04.822983 19310 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:04.824074 19310 fs_manager.cc:404] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:22:04.824255 19310 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:04.824364 19310 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:04.825188 19310 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:04.825584 19310 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal uuid: "d996c37c274e44238c372e99e3aaf130" format_stamp: "Formatted at 2019-11-21 10:22:00 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:04.825631 19310 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:04.851557 19310 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:04.851936 19310 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:04.851956 19310 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:04.857072 19310 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.4:46145 I1121 10:22:04.857385 19310 webserver.cc:382] Webserver started at http://127.5.70.4:33715/ using document root and password file I1121 10:22:04.857646 19310 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb I1121 10:22:04.861959 19431 heartbeater.cc:322] Connected to a master server at 127.5.70.62:44101 I1121 10:22:04.862004 19431 heartbeater.cc:406] Registering TS with master... I1121 10:22:04.862092 19431 heartbeater.cc:455] Master 127.5.70.62:44101 requested a full tablet report, sending... I1121 10:22:04.862390 18528 ts_manager.cc:192] Re-registered known tserver with Master: d996c37c274e44238c372e99e3aaf130 (127.5.70.4:46145) I1121 10:22:04.862969 18528 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.4:41539 I1121 10:22:04.863844 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 19310 Master Summary UUID | Address | Status ----------------------------------+-------------------+--------- b0d2cd8e977947beba82033b2b85722f | 127.5.70.62:44101 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status | Location ----------------------------------+------------------+---------+---------- 28e8d1b188ab48e993b0033564c2d09e | 127.5.70.2:42157 | HEALTHY | 3fcc96c2cb884d6b9c4f1ffa79564c59 | 127.5.70.1:44059 | HEALTHY | d996c37c274e44238c372e99e3aaf130 | 127.5.70.4:46145 | HEALTHY | e1619ca1896449d2bd8a7f68b8ece273 | 127.5.70.3:45069 | HEALTHY | Tablet Server Location Summary Location | Count ----------+--------- | 4 Flag | Value | Tags | Tablet Server ------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 4 server(s) checked local_ip_for_outbound_sockets | 127.5.70.1 | experimental | 127.5.70.1:44059 local_ip_for_outbound_sockets | 127.5.70.2 | experimental | 127.5.70.2:42157 local_ip_for_outbound_sockets | 127.5.70.3 | experimental | 127.5.70.3:45069 local_ip_for_outbound_sockets | 127.5.70.4 | experimental | 127.5.70.4:46145 never_fsync | true | unsafe,advanced | all 4 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked server_dump_info_format | pb | hidden | all 4 server(s) checked server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.70.1:44059 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.70.2:42157 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.70.3:45069 server_dump_info_path | /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.70.4:46145 Version Summary Version | Servers -----------------+------------------------- 1.12.0-SNAPSHOT | all 5 server(s) checked Tablet Summary Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ------------------------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0 rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 Tablet Replica Count Summary Statistic | Replica Count ----------------+--------------- Minimum | 0 First Quartile | 0 Median | 6 Third Quartile | 7 Maximum | 7 Total Count Summary | Total Count ----------------+------------- Masters | 1 Tablet Servers | 4 Tables | 5 Tablets | 13 Replicas | 13 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I1121 10:22:05.016089 5400 log_verifier.cc:121] Checking tablet 078d24c361bc4a668606a29cea4d9167 I1121 10:22:05.021450 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 078d24c361bc4a668606a29cea4d9167 I1121 10:22:05.021476 5400 log_verifier.cc:121] Checking tablet 6c2dbaff378a4bf1b768e12d054e5992 I1121 10:22:05.026182 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 6c2dbaff378a4bf1b768e12d054e5992 I1121 10:22:05.026209 5400 log_verifier.cc:121] Checking tablet 71102c6fcd874338bb12a326de2f66eb I1121 10:22:05.030841 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 71102c6fcd874338bb12a326de2f66eb I1121 10:22:05.030867 5400 log_verifier.cc:121] Checking tablet 73e224b39f574bfeab6ee341f1a161c6 I1121 10:22:05.035444 5400 log_verifier.cc:172] Verified matching terms for 2 ops in tablet 73e224b39f574bfeab6ee341f1a161c6 I1121 10:22:05.035470 5400 log_verifier.cc:121] Checking tablet 77515e9cab7d432c8a48ca4751056ebe I1121 10:22:05.040786 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 77515e9cab7d432c8a48ca4751056ebe I1121 10:22:05.040812 5400 log_verifier.cc:121] Checking tablet 851f425c2daf42d5a25e4894f2062ab6 I1121 10:22:05.046036 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 851f425c2daf42d5a25e4894f2062ab6 I1121 10:22:05.046078 5400 log_verifier.cc:121] Checking tablet 94fd9d56e2e84494b545ff7dfeeb2472 I1121 10:22:05.050981 5400 log_verifier.cc:172] Verified matching terms for 2 ops in tablet 94fd9d56e2e84494b545ff7dfeeb2472 I1121 10:22:05.051012 5400 log_verifier.cc:121] Checking tablet 978a93ad52404bba88734538acfacc4c I1121 10:22:05.056030 5400 log_verifier.cc:172] Verified matching terms for 2 ops in tablet 978a93ad52404bba88734538acfacc4c I1121 10:22:05.056056 5400 log_verifier.cc:121] Checking tablet b4e2be480c6f45d8827b04e898bfe164 I1121 10:22:05.060551 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet b4e2be480c6f45d8827b04e898bfe164 I1121 10:22:05.060572 5400 log_verifier.cc:121] Checking tablet b50bde701f684114a8a6694a0e7eccc5 I1121 10:22:05.065234 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet b50bde701f684114a8a6694a0e7eccc5 I1121 10:22:05.065259 5400 log_verifier.cc:121] Checking tablet bfc84ced32954427af051f3b0119983b I1121 10:22:05.069761 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet bfc84ced32954427af051f3b0119983b I1121 10:22:05.069783 5400 log_verifier.cc:121] Checking tablet dc66002863e7400a83880fb6283e77b6 I1121 10:22:05.074307 5400 log_verifier.cc:172] Verified matching terms for 1 ops in tablet dc66002863e7400a83880fb6283e77b6 I1121 10:22:05.074331 5400 log_verifier.cc:121] Checking tablet e6b46769540548608fc99e63af52cb12 I1121 10:22:05.079051 5400 log_verifier.cc:172] Verified matching terms for 2 ops in tablet e6b46769540548608fc99e63af52cb12 I1121 10:22:05.079476 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 18568 I1121 10:22:05.083258 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 18702 I1121 10:22:05.086685 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 19186 I1121 10:22:05.089759 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 19310 I1121 10:22:05.092600 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master with pid 18505 2019-11-21T10:22:05Z chronyd exiting [ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (5099 ms) [----------] 1 test from RebalancerAndSingleReplicaTablets (5099 ms total) [----------] 1 test from LocationAwareRebalancingParamTest [ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 I1121 10:22:05.118168 5400 test_util.cc:225] Using random seed: -1422132523 I1121 10:22:05.118571 5400 ts_itest-base.cc:112] Starting cluster with: I1121 10:22:05.118588 5400 ts_itest-base.cc:113] -------------- I1121 10:22:05.118592 5400 ts_itest-base.cc:114] 4 tablet servers I1121 10:22:05.118595 5400 ts_itest-base.cc:115] 2 replicas per TS I1121 10:22:05.118597 5400 ts_itest-base.cc:116] -------------- 2019-11-21T10:22:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG) 2019-11-21T10:22:05Z Disabled control of system clock I1121 10:22:05.125334 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:46063 --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --tserver_unresponsive_timeout_ms=3000 --raft_prepare_replacement_before_eviction=true --master_client_location_assignment_enabled=false --allow_unsafe_replication_factor --rpc_reuseport=true --location_mapping_cmd=/tmp/dist-test-taskdLJtk4/build/release/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1 --builtin_ntp_servers=127.5.70.20:34015 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:05.169286 19536 flags.cc:404] Enabled unsafe flag: --allow_unsafe_replication_factor=true W1121 10:22:05.169453 19536 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:05.169466 19536 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:05.170356 19536 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:05.170380 19536 flags.cc:404] Enabled experimental flag: --ipki_ca_key_size=1024 W1121 10:22:05.170389 19536 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:05.170394 19536 flags.cc:404] Enabled experimental flag: --tsk_num_rsa_bits=512 W1121 10:22:05.170400 19536 flags.cc:404] Enabled experimental flag: --rpc_reuseport=true I1121 10:22:05.171746 19536 master_main.cc:98] Master server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34015 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --location_mapping_cmd=/tmp/dist-test-taskdLJtk4/build/release/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1 --ipki_ca_key_size=1024 --master_client_location_assignment_enabled=false --tserver_unresponsive_timeout_ms=3000 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.70.62:46063 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.70.62 --webserver_port=0 --never_fsync=true --heap_profile_path=/tmp/kudu-master.19536 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:05.173557 19536 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:05.174700 19536 hybrid_clock.cc:183] HybridClock initialized: now 1574331725174679 us; error 38 us; skew 500 ppm I1121 10:22:05.174894 19536 fs_manager.cc:261] Metadata directory not provided I1121 10:22:05.174934 19536 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:05.175045 19536 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:05.175066 19536 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:05.176133 19536 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/instance: uuid: "ffce6515c23c4a1895d3dbb4d935f04a" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.176359 19536 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "ffce6515c23c4a1895d3dbb4d935f04a" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.177968 19536 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:22:05.178320 19536 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:05.178454 19536 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:05.179145 19536 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:05.179193 19536 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal uuid: "ffce6515c23c4a1895d3dbb4d935f04a" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.179258 19536 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:05.196458 19536 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:05.199394 19536 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.62:46063 I1121 10:22:05.199764 19536 webserver.cc:382] Webserver started at http://127.5.70.62:37913/ using document root and password file I1121 10:22:05.200057 19536 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/master-0/data/info.pb I1121 10:22:05.200783 19597 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:05.203322 19597 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a: Bootstrap starting. I1121 10:22:05.203821 19597 tablet_bootstrap.cc:590] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a: No blocks or log segments found. Creating new log. I1121 10:22:05.204084 19597 log.cc:810] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a: Log is configured to *not* fsync() on all Append() calls I1121 10:22:05.204632 19597 tablet_bootstrap.cc:439] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a: No bootstrap required, opened a new log I1121 10:22:05.206015 19597 raft_consensus.cc:337] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" member_type: VOTER } I1121 10:22:05.206060 19597 raft_consensus.cc:363] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:05.206073 19597 raft_consensus.cc:702] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffce6515c23c4a1895d3dbb4d935f04a, State: Initialized, Role: FOLLOWER I1121 10:22:05.206276 19597 consensus_queue.cc:231] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" member_type: VOTER } I1121 10:22:05.206318 19597 raft_consensus.cc:377] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I1121 10:22:05.206357 19599 raft_consensus.cc:469] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:05.206423 19599 raft_consensus.cc:491] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" member_type: VOTER } I1121 10:22:05.206487 19597 raft_consensus.cc:469] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I1121 10:22:05.206511 19597 raft_consensus.cc:2905] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:05.206526 19599 leader_election.cc:310] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffce6515c23c4a1895d3dbb4d935f04a; no voters: I1121 10:22:05.206636 19599 leader_election.cc:296] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers I1121 10:22:05.207015 19597 raft_consensus.cc:491] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" member_type: VOTER } I1121 10:22:05.207051 19597 leader_election.cc:310] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffce6515c23c4a1895d3dbb4d935f04a; no voters: I1121 10:22:05.207074 19597 leader_election.cc:296] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [CANDIDATE]: Term 1 election: Requested vote from peers I1121 10:22:05.207078 19599 raft_consensus.cc:2616] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won I1121 10:22:05.207106 19599 raft_consensus.cc:2653] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:05.207137 19599 raft_consensus.cc:664] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [term 1 LEADER]: Becoming Leader. State: Replica: ffce6515c23c4a1895d3dbb4d935f04a, State: Running, Role: LEADER I1121 10:22:05.207201 19599 consensus_queue.cc:208] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" member_type: VOTER } I1121 10:22:05.207486 19597 sys_catalog.cc:340] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" member_type: VOTER } } I1121 10:22:05.207518 19600 sys_catalog.cc:340] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" member_type: VOTER } } I1121 10:22:05.207522 19597 sys_catalog.cc:343] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [sys.catalog]: This master's current role is: LEADER I1121 10:22:05.207561 19601 sys_catalog.cc:340] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffce6515c23c4a1895d3dbb4d935f04a. Latest consensus state: current_term: 1 leader_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffce6515c23c4a1895d3dbb4d935f04a" member_type: VOTER } } I1121 10:22:05.207547 19600 sys_catalog.cc:343] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [sys.catalog]: This master's current role is: LEADER I1121 10:22:05.207592 19601 sys_catalog.cc:343] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [sys.catalog]: This master's current role is: LEADER I1121 10:22:05.207718 19597 sys_catalog.cc:428] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a [sys.catalog]: configured and running, proceeding with master startup. W1121 10:22:05.208492 19604 catalog_manager.cc:1159] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a: acquiring CA information for follower catalog manager: Not found: root CA entry not found I1121 10:22:05.208624 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master as pid 19536 I1121 10:22:05.209018 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.70.1:0 --local_ip_for_outbound_sockets=127.5.70.1 --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:46063 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34015 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} W1121 10:22:05.212548 19604 catalog_manager.cc:631] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I1121 10:22:05.212617 19603 catalog_manager.cc:1090] Loading table and tablet metadata into memory... I1121 10:22:05.212777 19603 catalog_manager.cc:1101] Initializing Kudu internal certificate authority... I1121 10:22:05.224910 19603 catalog_manager.cc:985] Generated new certificate authority record I1121 10:22:05.225625 19603 catalog_manager.cc:1110] Loading token signing keys... I1121 10:22:05.228555 19603 catalog_manager.cc:4455] T 00000000000000000000000000000000 P ffce6515c23c4a1895d3dbb4d935f04a: Generated new TSK 0 I1121 10:22:05.228670 19603 catalog_manager.cc:1120] Initializing in-progress tserver states... WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:05.264326 19607 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:05.264564 19607 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:05.265713 19607 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:05.265743 19607 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:05.265772 19607 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.1 I1121 10:22:05.267453 19607 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34015 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.1:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.70.1 --webserver_port=0 --tserver_master_addrs=127.5.70.62:46063 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.19607 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:05.270041 19607 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:05.277400 19607 hybrid_clock.cc:183] HybridClock initialized: now 1574331725277355 us; error 55 us; skew 500 ppm I1121 10:22:05.277616 19607 fs_manager.cc:261] Metadata directory not provided I1121 10:22:05.277669 19607 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:05.277801 19607 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:05.277830 19607 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:05.279296 19607 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "8754ab0c37934104af1213092861d2fa" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.279526 19607 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "8754ab0c37934104af1213092861d2fa" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.282122 19607 fs_manager.cc:508] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s I1121 10:22:05.282485 19607 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:05.282634 19607 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:05.283524 19607 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:05.283569 19607 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal uuid: "8754ab0c37934104af1213092861d2fa" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.283620 19607 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:05.330958 19607 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:05.331411 19607 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:05.331436 19607 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:05.337394 19607 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.1:33643 I1121 10:22:05.337757 19607 webserver.cc:382] Webserver started at http://127.5.70.1:43315/ using document root and password file I1121 10:22:05.338032 19607 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/data/info.pb I1121 10:22:05.343343 19729 heartbeater.cc:322] Connected to a master server at 127.5.70.62:46063 I1121 10:22:05.343396 19729 heartbeater.cc:406] Registering TS with master... I1121 10:22:05.343504 19729 heartbeater.cc:455] Master 127.5.70.62:46063 requested a full tablet report, sending... I1121 10:22:05.344770 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 19607 I1121 10:22:05.345074 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.70.2:0 --local_ip_for_outbound_sockets=127.5.70.2 --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:46063 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34015 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:22:05.389050 19564 ts_manager.cc:192] Registered new tserver with Master: 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:05.390043 19564 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.1:42331 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:05.396015 19734 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:05.396154 19734 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:05.396996 19734 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:05.397017 19734 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:05.397032 19734 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.2 I1121 10:22:05.398183 19734 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34015 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.2:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.70.2 --webserver_port=0 --tserver_master_addrs=127.5.70.62:46063 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.19734 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.2 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:05.400007 19734 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:05.401141 19734 hybrid_clock.cc:183] HybridClock initialized: now 1574331725401123 us; error 38 us; skew 500 ppm I1121 10:22:05.401322 19734 fs_manager.cc:261] Metadata directory not provided I1121 10:22:05.401360 19734 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:05.401463 19734 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:05.401485 19734 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:05.402506 19734 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "389dbaf174a9403dbe2a244219e0dbb2" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.402698 19734 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "389dbaf174a9403dbe2a244219e0dbb2" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.404145 19734 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s I1121 10:22:05.404443 19734 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:05.404598 19734 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:05.405344 19734 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:05.405383 19734 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal uuid: "389dbaf174a9403dbe2a244219e0dbb2" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.405434 19734 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:05.420140 19734 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:05.420555 19734 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:05.420576 19734 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:05.426075 19734 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.2:36033 I1121 10:22:05.426400 19734 webserver.cc:382] Webserver started at http://127.5.70.2:45307/ using document root and password file I1121 10:22:05.426642 19734 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb I1121 10:22:05.429759 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 19734 I1121 10:22:05.430037 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.70.3:0 --local_ip_for_outbound_sockets=127.5.70.3 --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:46063 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34015 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:22:05.430981 19855 heartbeater.cc:322] Connected to a master server at 127.5.70.62:46063 I1121 10:22:05.431039 19855 heartbeater.cc:406] Registering TS with master... I1121 10:22:05.431164 19855 heartbeater.cc:455] Master 127.5.70.62:46063 requested a full tablet report, sending... I1121 10:22:05.486768 19564 ts_manager.cc:192] Registered new tserver with Master: 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2:36033) I1121 10:22:05.487393 19564 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.2:54381 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:05.495880 19858 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:05.496031 19858 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:05.496913 19858 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:05.496934 19858 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:05.496953 19858 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.3 I1121 10:22:05.498124 19858 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34015 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.3:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.70.3 --webserver_port=0 --tserver_master_addrs=127.5.70.62:46063 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.19858 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.3 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:05.499919 19858 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:05.501062 19858 hybrid_clock.cc:183] HybridClock initialized: now 1574331725501038 us; error 36 us; skew 500 ppm I1121 10:22:05.501183 19858 fs_manager.cc:261] Metadata directory not provided I1121 10:22:05.501211 19858 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:05.501294 19858 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:05.501312 19858 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:05.502358 19858 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "9a5dc7ff276144dfb28b69a0dd92218d" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.502526 19858 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "9a5dc7ff276144dfb28b69a0dd92218d" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.503872 19858 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:22:05.504113 19858 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:05.504223 19858 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:05.505012 19858 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:05.505048 19858 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal uuid: "9a5dc7ff276144dfb28b69a0dd92218d" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.505081 19858 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:05.529480 19858 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:05.529850 19858 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:05.529867 19858 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:05.534323 19858 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.3:36993 I1121 10:22:05.534627 19858 webserver.cc:382] Webserver started at http://127.5.70.3:44583/ using document root and password file I1121 10:22:05.534868 19858 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb I1121 10:22:05.535882 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 19858 I1121 10:22:05.536186 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.70.4:0 --local_ip_for_outbound_sockets=127.5.70.4 --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:46063 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34015 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} I1121 10:22:05.539803 19980 heartbeater.cc:322] Connected to a master server at 127.5.70.62:46063 I1121 10:22:05.539863 19980 heartbeater.cc:406] Registering TS with master... I1121 10:22:05.539980 19980 heartbeater.cc:455] Master 127.5.70.62:46063 requested a full tablet report, sending... I1121 10:22:05.594266 19564 ts_manager.cc:192] Registered new tserver with Master: 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3:36993) I1121 10:22:05.594913 19564 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.3:37975 WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:05.600824 19983 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:05.600962 19983 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:05.601727 19983 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:05.601748 19983 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:05.601766 19983 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.4 I1121 10:22:05.602917 19983 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34015 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.4:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.70.4 --webserver_port=0 --tserver_master_addrs=127.5.70.62:46063 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.19983 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.4 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:05.604723 19983 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:05.605891 19983 hybrid_clock.cc:183] HybridClock initialized: now 1574331725605860 us; error 37 us; skew 500 ppm I1121 10:22:05.606045 19983 fs_manager.cc:261] Metadata directory not provided I1121 10:22:05.606076 19983 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:05.606158 19983 server_base.cc:452] Could not load existing FS layout: Not found: could not find a healthy instance file I1121 10:22:05.606175 19983 server_base.cc:453] Attempting to create new FS layout instead I1121 10:22:05.607197 19983 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "4a73706e30e7445a93af7bd060d6103b" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.607365 19983 fs_manager.cc:607] Generated new instance metadata in path /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "4a73706e30e7445a93af7bd060d6103b" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.608732 19983 fs_manager.cc:508] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s I1121 10:22:05.608979 19983 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:05.609072 19983 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:05.609786 19983 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:05.609827 19983 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal uuid: "4a73706e30e7445a93af7bd060d6103b" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:05.609890 19983 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:05.625257 19983 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:05.625596 19983 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:05.625615 19983 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:05.630812 19983 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.4:44823 I1121 10:22:05.631131 19983 webserver.cc:382] Webserver started at http://127.5.70.4:38831/ using document root and password file I1121 10:22:05.631378 19983 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/data/info.pb I1121 10:22:05.635681 20105 heartbeater.cc:322] Connected to a master server at 127.5.70.62:46063 I1121 10:22:05.635771 20105 heartbeater.cc:406] Registering TS with master... I1121 10:22:05.635860 20105 heartbeater.cc:455] Master 127.5.70.62:46063 requested a full tablet report, sending... I1121 10:22:05.640889 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 19983 I1121 10:22:05.665238 19564 ts_manager.cc:192] Registered new tserver with Master: 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:05.665798 19564 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.4:38631 I1121 10:22:05.666594 5400 external_mini_cluster.cc:717] 4 TS(s) registered with all masters I1121 10:22:05.671097 5400 external_mini_cluster.cc:717] 4 TS(s) registered with all masters I1121 10:22:05.671169 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 19734 I1121 10:22:05.674437 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 19858 I1121 10:22:06.391095 19729 heartbeater.cc:447] Master 127.5.70.62:46063 was elected leader, sending a full tablet report... I1121 10:22:06.666762 20105 heartbeater.cc:447] Master 127.5.70.62:46063 was elected leader, sending a full tablet report... I1121 10:22:09.429601 19564 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42720: name: "rebalance_test_table_0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } W1121 10:22:09.429800 19564 catalog_manager.cc:1620] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers. I1121 10:22:09.435860 20041 tablet_service.cc:1023] Processing CreateTablet for tablet fd080f03ef3d4168bfb86bc89b3b4c18 (table=rebalance_test_table_0 [id=dca22bab1f104febb0f86c11e01c69b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.435914 20042 tablet_service.cc:1023] Processing CreateTablet for tablet b6bcafd84c0949b6b7d7fa7dc447f620 (table=rebalance_test_table_0 [id=dca22bab1f104febb0f86c11e01c69b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.436097 20041 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet fd080f03ef3d4168bfb86bc89b3b4c18. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.436215 20042 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b6bcafd84c0949b6b7d7fa7dc447f620. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.436278 19657 tablet_service.cc:1023] Processing CreateTablet for tablet 74d663e39077406590cb0c9d4204ac1d (table=rebalance_test_table_0 [id=dca22bab1f104febb0f86c11e01c69b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.436293 19649 tablet_service.cc:1023] Processing CreateTablet for tablet b6bcafd84c0949b6b7d7fa7dc447f620 (table=rebalance_test_table_0 [id=dca22bab1f104febb0f86c11e01c69b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.436396 19650 tablet_service.cc:1023] Processing CreateTablet for tablet fd080f03ef3d4168bfb86bc89b3b4c18 (table=rebalance_test_table_0 [id=dca22bab1f104febb0f86c11e01c69b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.436465 19657 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 74d663e39077406590cb0c9d4204ac1d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.436594 19649 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b6bcafd84c0949b6b7d7fa7dc447f620. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.436805 19650 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet fd080f03ef3d4168bfb86bc89b3b4c18. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.438632 20123 tablet_bootstrap.cc:439] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: Bootstrap starting. I1121 10:22:09.439211 20123 tablet_bootstrap.cc:590] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: No blocks or log segments found. Creating new log. I1121 10:22:09.439442 20123 log.cc:810] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: Log is configured to *not* fsync() on all Append() calls I1121 10:22:09.439859 20124 tablet_bootstrap.cc:439] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa: Bootstrap starting. I1121 10:22:09.440023 20040 tablet_service.cc:1023] Processing CreateTablet for tablet 74d663e39077406590cb0c9d4204ac1d (table=rebalance_test_table_0 [id=dca22bab1f104febb0f86c11e01c69b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.440290 20040 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 74d663e39077406590cb0c9d4204ac1d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.440430 20124 tablet_bootstrap.cc:590] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa: No blocks or log segments found. Creating new log. I1121 10:22:09.440752 20124 log.cc:810] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa: Log is configured to *not* fsync() on all Append() calls I1121 10:22:09.441319 20124 tablet_bootstrap.cc:439] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa: No bootstrap required, opened a new log I1121 10:22:09.441375 20124 ts_tablet_manager.cc:1104] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:22:09.441545 20123 tablet_bootstrap.cc:439] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: No bootstrap required, opened a new log I1121 10:22:09.441594 20123 ts_tablet_manager.cc:1104] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s I1121 10:22:09.442646 20124 raft_consensus.cc:337] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.442661 20123 raft_consensus.cc:337] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.442687 20124 raft_consensus.cc:363] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.442708 20124 raft_consensus.cc:702] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Initialized, Role: FOLLOWER I1121 10:22:09.442714 20123 raft_consensus.cc:363] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.442739 20123 raft_consensus.cc:702] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Initialized, Role: FOLLOWER I1121 10:22:09.442857 20123 consensus_queue.cc:231] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.442857 20124 consensus_queue.cc:231] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.443029 20124 ts_tablet_manager.cc:1132] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I1121 10:22:09.443029 20123 ts_tablet_manager.cc:1132] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s I1121 10:22:09.443084 20123 tablet_bootstrap.cc:439] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b: Bootstrap starting. I1121 10:22:09.443085 20124 tablet_bootstrap.cc:439] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa: Bootstrap starting. I1121 10:22:09.443398 20124 tablet_bootstrap.cc:590] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa: No blocks or log segments found. Creating new log. I1121 10:22:09.443471 20123 tablet_bootstrap.cc:590] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b: No blocks or log segments found. Creating new log. I1121 10:22:09.444252 20124 tablet_bootstrap.cc:439] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa: No bootstrap required, opened a new log I1121 10:22:09.444289 20124 ts_tablet_manager.cc:1104] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:09.444391 20123 tablet_bootstrap.cc:439] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b: No bootstrap required, opened a new log I1121 10:22:09.444424 20123 ts_tablet_manager.cc:1104] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:09.444433 20124 raft_consensus.cc:337] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.444447 20124 raft_consensus.cc:363] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.444455 20124 raft_consensus.cc:702] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Initialized, Role: FOLLOWER I1121 10:22:09.444542 20124 consensus_queue.cc:231] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.444600 20123 raft_consensus.cc:337] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.444623 20123 raft_consensus.cc:363] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.444630 20123 raft_consensus.cc:702] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Initialized, Role: FOLLOWER I1121 10:22:09.444723 20123 consensus_queue.cc:231] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.444789 20124 ts_tablet_manager.cc:1132] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:09.444841 20124 tablet_bootstrap.cc:439] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: Bootstrap starting. I1121 10:22:09.445088 20123 ts_tablet_manager.cc:1132] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:22:09.445140 20123 tablet_bootstrap.cc:439] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b: Bootstrap starting. I1121 10:22:09.445160 20124 tablet_bootstrap.cc:590] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: No blocks or log segments found. Creating new log. I1121 10:22:09.445511 20123 tablet_bootstrap.cc:590] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b: No blocks or log segments found. Creating new log. I1121 10:22:09.446055 20124 tablet_bootstrap.cc:439] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: No bootstrap required, opened a new log I1121 10:22:09.446108 20124 ts_tablet_manager.cc:1104] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:09.446146 20123 tablet_bootstrap.cc:439] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b: No bootstrap required, opened a new log I1121 10:22:09.446187 20123 ts_tablet_manager.cc:1104] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:09.446318 20124 raft_consensus.cc:337] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.446333 20123 raft_consensus.cc:337] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.446346 20123 raft_consensus.cc:363] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.446346 20124 raft_consensus.cc:363] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.446352 20123 raft_consensus.cc:702] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Initialized, Role: FOLLOWER I1121 10:22:09.446353 20124 raft_consensus.cc:702] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Initialized, Role: FOLLOWER I1121 10:22:09.446432 20124 consensus_queue.cc:231] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.446432 20123 consensus_queue.cc:231] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.446533 20124 ts_tablet_manager.cc:1132] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s I1121 10:22:09.446682 20123 ts_tablet_manager.cc:1132] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s W1121 10:22:09.590999 19730 tablet.cc:1767] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:22:09.604043 20128 raft_consensus.cc:469] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:09.604200 20128 raft_consensus.cc:491] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.604436 20128 leader_election.cc:296] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:09.607331 19681 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b6bcafd84c0949b6b7d7fa7dc447f620" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" is_pre_election: true I1121 10:22:09.607393 19681 raft_consensus.cc:2359] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 0. I1121 10:22:09.607619 19995 leader_election.cc:310] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:09.607719 20128 raft_consensus.cc:2653] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:22:09.607797 20128 raft_consensus.cc:469] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:22:09.607808 20128 raft_consensus.cc:2905] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:09.608494 20128 raft_consensus.cc:491] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.608600 20128 leader_election.cc:296] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Requested vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:09.608767 19681 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b6bcafd84c0949b6b7d7fa7dc447f620" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" I1121 10:22:09.608798 19681 raft_consensus.cc:2905] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:09.609357 19681 raft_consensus.cc:2359] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 1. I1121 10:22:09.609549 19995 leader_election.cc:310] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:09.609643 20128 raft_consensus.cc:2653] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:09.609746 20128 raft_consensus.cc:664] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Becoming Leader. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Running, Role: LEADER I1121 10:22:09.609838 20128 consensus_queue.cc:208] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.610581 19563 catalog_manager.cc:4143] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: term changed from 0 to 1, leader changed from to 4a73706e30e7445a93af7bd060d6103b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } } W1121 10:22:09.634074 20106 tablet.cc:1767] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b: Can't schedule compaction. Clean time has not been advanced past its initial value. I1121 10:22:09.714530 20128 raft_consensus.cc:469] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:09.714619 20128 raft_consensus.cc:491] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.714690 20128 leader_election.cc:296] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:09.714990 19681 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "74d663e39077406590cb0c9d4204ac1d" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" is_pre_election: true I1121 10:22:09.715034 19681 raft_consensus.cc:2359] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 0. I1121 10:22:09.715222 19995 leader_election.cc:310] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:09.715312 20128 raft_consensus.cc:2653] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:22:09.715334 20128 raft_consensus.cc:469] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:22:09.715339 20128 raft_consensus.cc:2905] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:09.715952 20128 raft_consensus.cc:491] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.716007 20128 leader_election.cc:296] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Requested vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:09.716279 19681 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "74d663e39077406590cb0c9d4204ac1d" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" I1121 10:22:09.716305 19681 raft_consensus.cc:2905] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:09.716848 19681 raft_consensus.cc:2359] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 1. I1121 10:22:09.717028 19995 leader_election.cc:310] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:09.717103 20128 raft_consensus.cc:2653] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:09.717136 20128 raft_consensus.cc:664] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Becoming Leader. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Running, Role: LEADER I1121 10:22:09.717183 20128 consensus_queue.cc:208] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.717918 19563 catalog_manager.cc:4143] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b reported cstate change: term changed from 0 to 1, leader changed from to 4a73706e30e7445a93af7bd060d6103b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } } I1121 10:22:09.866142 20128 raft_consensus.cc:469] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:09.866294 20128 raft_consensus.cc:491] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.866376 20128 leader_election.cc:296] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:09.866621 20127 raft_consensus.cc:469] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:09.866652 19681 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fd080f03ef3d4168bfb86bc89b3b4c18" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" is_pre_election: true I1121 10:22:09.866703 20127 raft_consensus.cc:491] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.866818 19681 raft_consensus.cc:2359] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 0. I1121 10:22:09.866945 20127 leader_election.cc:296] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:09.866992 19995 leader_election.cc:310] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:09.867084 20128 raft_consensus.cc:2653] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:22:09.867100 20128 raft_consensus.cc:469] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:22:09.867107 20128 raft_consensus.cc:2905] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:09.867993 20128 raft_consensus.cc:491] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.868046 20128 leader_election.cc:296] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Requested vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:09.868201 19681 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fd080f03ef3d4168bfb86bc89b3b4c18" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" I1121 10:22:09.868235 19681 raft_consensus.cc:2905] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:09.868782 19681 raft_consensus.cc:2359] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 1. I1121 10:22:09.868921 19995 leader_election.cc:310] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:09.869004 20128 raft_consensus.cc:2653] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:09.869058 20128 raft_consensus.cc:664] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Becoming Leader. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Running, Role: LEADER I1121 10:22:09.869149 20128 consensus_queue.cc:208] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.869793 19563 catalog_manager.cc:4143] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: term changed from 0 to 1, leader changed from to 4a73706e30e7445a93af7bd060d6103b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } } I1121 10:22:09.870064 20062 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "fd080f03ef3d4168bfb86bc89b3b4c18" candidate_uuid: "8754ab0c37934104af1213092861d2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a73706e30e7445a93af7bd060d6103b" is_pre_election: true I1121 10:22:09.870110 20062 raft_consensus.cc:2320] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 8754ab0c37934104af1213092861d2fa for term 1 because replica is either leader or believes a valid leader to be alive. I1121 10:22:09.870517 19618 leader_election.cc:310] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8754ab0c37934104af1213092861d2fa; no voters: 4a73706e30e7445a93af7bd060d6103b I1121 10:22:09.870599 20127 raft_consensus.cc:2601] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority W1121 10:22:09.929539 20139 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_0' I1121 10:22:09.961596 19681 raft_consensus.cc:1181] T fd080f03ef3d4168bfb86bc89b3b4c18 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:22:09.961644 19679 raft_consensus.cc:1181] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:22:09.961884 20128 consensus_queue.cc:984] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:22:09.961953 19679 raft_consensus.cc:1181] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:22:09.962142 20128 consensus_queue.cc:984] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:22:09.962344 20128 consensus_queue.cc:984] T fd080f03ef3d4168bfb86bc89b3b4c18 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:22:09.966236 20158 mvcc.cc:203] Tried to move safe_time back from 6448462765920915456 to 6448462764921790464. Current Snapshot: MvccSnapshot[committed={T|T < 6448462765920915456}] I1121 10:22:09.966854 20155 mvcc.cc:203] Tried to move safe_time back from 6448462765922656256 to 6448462764482162688. Current Snapshot: MvccSnapshot[committed={T|T < 6448462765922656256}] Generator report time total : 24.7433 ms time per row: 0.0123717 ms I1121 10:22:09.976730 19563 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42720: name: "rebalance_test_table_1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } W1121 10:22:09.976867 19563 catalog_manager.cc:1620] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers. I1121 10:22:09.978355 20040 tablet_service.cc:1023] Processing CreateTablet for tablet aefc5ffe5279415b858715c9e0e997c7 (table=rebalance_test_table_1 [id=7cd596ea2e5c4cdb9985bbd0788e6f8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.978391 19657 tablet_service.cc:1023] Processing CreateTablet for tablet aefc5ffe5279415b858715c9e0e997c7 (table=rebalance_test_table_1 [id=7cd596ea2e5c4cdb9985bbd0788e6f8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.978435 19650 tablet_service.cc:1023] Processing CreateTablet for tablet 0d267eb692ae4744a5d2d38265f30121 (table=rebalance_test_table_1 [id=7cd596ea2e5c4cdb9985bbd0788e6f8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.978480 20040 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet aefc5ffe5279415b858715c9e0e997c7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.978499 19657 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet aefc5ffe5279415b858715c9e0e997c7. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.978529 19649 tablet_service.cc:1023] Processing CreateTablet for tablet 81897d1f2f904d2e89095a4818043edf (table=rebalance_test_table_1 [id=7cd596ea2e5c4cdb9985bbd0788e6f8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.978572 20041 tablet_service.cc:1023] Processing CreateTablet for tablet 0d267eb692ae4744a5d2d38265f30121 (table=rebalance_test_table_1 [id=7cd596ea2e5c4cdb9985bbd0788e6f8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.978588 19650 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0d267eb692ae4744a5d2d38265f30121. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.978649 20041 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0d267eb692ae4744a5d2d38265f30121. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.979195 19649 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 81897d1f2f904d2e89095a4818043edf. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.979730 20042 tablet_service.cc:1023] Processing CreateTablet for tablet 81897d1f2f904d2e89095a4818043edf (table=rebalance_test_table_1 [id=7cd596ea2e5c4cdb9985bbd0788e6f8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:09.979807 20042 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 81897d1f2f904d2e89095a4818043edf. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:09.980342 20174 tablet_bootstrap.cc:439] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b: Bootstrap starting. I1121 10:22:09.980744 20174 tablet_bootstrap.cc:590] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b: No blocks or log segments found. Creating new log. I1121 10:22:09.981060 20175 tablet_bootstrap.cc:439] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa: Bootstrap starting. I1121 10:22:09.981355 20175 tablet_bootstrap.cc:590] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa: No blocks or log segments found. Creating new log. I1121 10:22:09.981798 20174 tablet_bootstrap.cc:439] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b: No bootstrap required, opened a new log I1121 10:22:09.981817 20175 tablet_bootstrap.cc:439] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa: No bootstrap required, opened a new log I1121 10:22:09.981842 20175 ts_tablet_manager.cc:1104] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:09.981839 20174 ts_tablet_manager.cc:1104] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s I1121 10:22:09.981962 20174 raft_consensus.cc:337] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.981963 20175 raft_consensus.cc:337] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.981982 20175 raft_consensus.cc:363] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.981983 20174 raft_consensus.cc:363] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.981987 20175 raft_consensus.cc:702] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Initialized, Role: FOLLOWER I1121 10:22:09.981992 20174 raft_consensus.cc:702] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Initialized, Role: FOLLOWER I1121 10:22:09.982043 20175 consensus_queue.cc:231] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.982102 20174 consensus_queue.cc:231] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.982127 20175 ts_tablet_manager.cc:1132] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:09.982163 20175 tablet_bootstrap.cc:439] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa: Bootstrap starting. I1121 10:22:09.982313 20174 ts_tablet_manager.cc:1132] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:09.982406 20174 tablet_bootstrap.cc:439] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b: Bootstrap starting. I1121 10:22:09.982617 20175 tablet_bootstrap.cc:590] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa: No blocks or log segments found. Creating new log. I1121 10:22:09.982704 20174 tablet_bootstrap.cc:590] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b: No blocks or log segments found. Creating new log. I1121 10:22:09.983202 20174 tablet_bootstrap.cc:439] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b: No bootstrap required, opened a new log I1121 10:22:09.983242 20174 ts_tablet_manager.cc:1104] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:09.983361 20174 raft_consensus.cc:337] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.983384 20174 raft_consensus.cc:363] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.983392 20174 raft_consensus.cc:702] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Initialized, Role: FOLLOWER I1121 10:22:09.983475 20174 consensus_queue.cc:231] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:09.983551 20175 tablet_bootstrap.cc:439] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa: No bootstrap required, opened a new log I1121 10:22:09.983588 20175 ts_tablet_manager.cc:1104] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:09.983613 20174 ts_tablet_manager.cc:1132] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:09.983645 20174 tablet_bootstrap.cc:439] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b: Bootstrap starting. I1121 10:22:09.983898 20174 tablet_bootstrap.cc:590] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b: No blocks or log segments found. Creating new log. I1121 10:22:09.984059 20175 raft_consensus.cc:337] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.984086 20175 raft_consensus.cc:363] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.984092 20175 raft_consensus.cc:702] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Initialized, Role: FOLLOWER I1121 10:22:09.984182 20175 consensus_queue.cc:231] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.984473 20175 ts_tablet_manager.cc:1132] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I1121 10:22:09.984614 20174 tablet_bootstrap.cc:439] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b: No bootstrap required, opened a new log I1121 10:22:09.984555 20175 tablet_bootstrap.cc:439] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa: Bootstrap starting. I1121 10:22:09.984654 20174 ts_tablet_manager.cc:1104] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:09.984797 20174 raft_consensus.cc:337] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.984822 20174 raft_consensus.cc:363] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.984828 20174 raft_consensus.cc:702] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Initialized, Role: FOLLOWER I1121 10:22:09.984895 20174 consensus_queue.cc:231] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.985078 20175 tablet_bootstrap.cc:590] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa: No blocks or log segments found. Creating new log. I1121 10:22:09.985157 20174 ts_tablet_manager.cc:1132] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:09.985775 20175 tablet_bootstrap.cc:439] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa: No bootstrap required, opened a new log I1121 10:22:09.985807 20175 ts_tablet_manager.cc:1104] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:09.985944 20175 raft_consensus.cc:337] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.985963 20175 raft_consensus.cc:363] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:09.985970 20175 raft_consensus.cc:702] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Initialized, Role: FOLLOWER I1121 10:22:09.986027 20175 consensus_queue.cc:231] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.986096 20175 ts_tablet_manager.cc:1132] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:09.989385 20157 raft_consensus.cc:469] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:09.989446 20157 raft_consensus.cc:491] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.989504 20157 leader_election.cc:296] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:09.989691 20062 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "81897d1f2f904d2e89095a4818043edf" candidate_uuid: "8754ab0c37934104af1213092861d2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a73706e30e7445a93af7bd060d6103b" is_pre_election: true I1121 10:22:09.989784 20062 raft_consensus.cc:2359] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8754ab0c37934104af1213092861d2fa in term 0. I1121 10:22:09.989933 19618 leader_election.cc:310] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:09.990012 20157 raft_consensus.cc:2653] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:22:09.990037 20157 raft_consensus.cc:469] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:22:09.990046 20157 raft_consensus.cc:2905] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:09.990592 20157 raft_consensus.cc:491] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.990694 20157 leader_election.cc:296] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 election: Requested vote from peers 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:09.990962 20062 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "81897d1f2f904d2e89095a4818043edf" candidate_uuid: "8754ab0c37934104af1213092861d2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a73706e30e7445a93af7bd060d6103b" I1121 10:22:09.990993 20062 raft_consensus.cc:2905] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:09.991539 20062 raft_consensus.cc:2359] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8754ab0c37934104af1213092861d2fa in term 1. I1121 10:22:09.991724 19618 leader_election.cc:310] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:09.991799 20157 raft_consensus.cc:2653] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:09.991847 20157 raft_consensus.cc:664] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 1 LEADER]: Becoming Leader. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Running, Role: LEADER I1121 10:22:09.991978 20157 consensus_queue.cc:208] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:09.992902 19563 catalog_manager.cc:4143] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa reported cstate change: term changed from 0 to 1, leader changed from to 8754ab0c37934104af1213092861d2fa (127.5.70.1). New cstate: current_term: 1 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: HEALTHY } } } I1121 10:22:10.037338 20167 raft_consensus.cc:469] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:10.037449 20167 raft_consensus.cc:491] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.037581 20167 leader_election.cc:296] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:10.037845 19679 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "aefc5ffe5279415b858715c9e0e997c7" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" is_pre_election: true I1121 10:22:10.037891 19679 raft_consensus.cc:2359] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 0. I1121 10:22:10.038053 19995 leader_election.cc:310] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.038161 20167 raft_consensus.cc:2653] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:22:10.038188 20167 raft_consensus.cc:469] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:22:10.038195 20167 raft_consensus.cc:2905] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.038990 20167 raft_consensus.cc:491] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.039047 20167 leader_election.cc:296] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Requested vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:10.039255 19679 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "aefc5ffe5279415b858715c9e0e997c7" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" I1121 10:22:10.039283 19679 raft_consensus.cc:2905] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.039837 19679 raft_consensus.cc:2359] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 1. I1121 10:22:10.039991 19995 leader_election.cc:310] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.040078 20167 raft_consensus.cc:2653] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:10.040127 20167 raft_consensus.cc:664] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Becoming Leader. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Running, Role: LEADER I1121 10:22:10.040190 20167 consensus_queue.cc:208] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.040835 19563 catalog_manager.cc:4143] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: term changed from 0 to 1, leader changed from to 4a73706e30e7445a93af7bd060d6103b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: UNKNOWN } } } I1121 10:22:10.099982 20167 raft_consensus.cc:469] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:10.100076 20167 raft_consensus.cc:491] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.100172 20167 leader_election.cc:296] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:10.100466 19679 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0d267eb692ae4744a5d2d38265f30121" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" is_pre_election: true I1121 10:22:10.100565 19679 raft_consensus.cc:2359] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 0. I1121 10:22:10.100723 19995 leader_election.cc:310] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.100808 20167 raft_consensus.cc:2653] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:22:10.100832 20167 raft_consensus.cc:469] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:22:10.100847 20167 raft_consensus.cc:2905] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.101531 20167 raft_consensus.cc:491] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.101586 20167 leader_election.cc:296] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Requested vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:10.101840 19679 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "0d267eb692ae4744a5d2d38265f30121" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" I1121 10:22:10.101866 19679 raft_consensus.cc:2905] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.102361 19679 raft_consensus.cc:2359] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 1. I1121 10:22:10.102521 19995 leader_election.cc:310] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.102588 20167 raft_consensus.cc:2653] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:10.102623 20167 raft_consensus.cc:664] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Becoming Leader. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Running, Role: LEADER I1121 10:22:10.102686 20167 consensus_queue.cc:208] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.103332 19563 catalog_manager.cc:4143] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: term changed from 0 to 1, leader changed from to 4a73706e30e7445a93af7bd060d6103b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } } W1121 10:22:10.180299 20185 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_1' I1121 10:22:10.204303 19679 raft_consensus.cc:1181] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:22:10.204371 19681 raft_consensus.cc:1181] T aefc5ffe5279415b858715c9e0e997c7 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:22:10.204805 20165 consensus_queue.cc:984] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:22:10.204919 20165 consensus_queue.cc:984] T aefc5ffe5279415b858715c9e0e997c7 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:22:10.205904 20062 raft_consensus.cc:1181] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:22:10.206130 20157 consensus_queue.cc:984] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 15.9832 ms time per row: 0.00799161 ms I1121 10:22:10.215584 19556 catalog_manager.cc:1455] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42720: name: "rebalance_test_table_2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } W1121 10:22:10.215695 19556 catalog_manager.cc:1620] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers. I1121 10:22:10.217087 20042 tablet_service.cc:1023] Processing CreateTablet for tablet 8f0b8462a15040e88aebbbf5d5bf9dcd (table=rebalance_test_table_2 [id=302bc66d906a41448959053fb1bde011]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:10.217125 20041 tablet_service.cc:1023] Processing CreateTablet for tablet 5ab68244024a4864b50a76748a5a6c22 (table=rebalance_test_table_2 [id=302bc66d906a41448959053fb1bde011]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:10.217139 19650 tablet_service.cc:1023] Processing CreateTablet for tablet 5ab68244024a4864b50a76748a5a6c22 (table=rebalance_test_table_2 [id=302bc66d906a41448959053fb1bde011]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I1121 10:22:10.217186 19649 tablet_service.cc:1023] Processing CreateTablet for tablet 99f1a040116d4b36b79b74919f086440 (table=rebalance_test_table_2 [id=302bc66d906a41448959053fb1bde011]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:10.217226 20042 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8f0b8462a15040e88aebbbf5d5bf9dcd. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:10.217239 19650 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5ab68244024a4864b50a76748a5a6c22. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:10.217267 20040 tablet_service.cc:1023] Processing CreateTablet for tablet 99f1a040116d4b36b79b74919f086440 (table=rebalance_test_table_2 [id=302bc66d906a41448959053fb1bde011]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I1121 10:22:10.217146 19657 tablet_service.cc:1023] Processing CreateTablet for tablet 8f0b8462a15040e88aebbbf5d5bf9dcd (table=rebalance_test_table_2 [id=302bc66d906a41448959053fb1bde011]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I1121 10:22:10.217278 19649 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 99f1a040116d4b36b79b74919f086440. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:10.217326 20041 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5ab68244024a4864b50a76748a5a6c22. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:10.217329 19657 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8f0b8462a15040e88aebbbf5d5bf9dcd. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:10.217770 20040 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 99f1a040116d4b36b79b74919f086440. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:10.218400 20175 tablet_bootstrap.cc:439] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa: Bootstrap starting. I1121 10:22:10.218678 20175 tablet_bootstrap.cc:590] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa: No blocks or log segments found. Creating new log. I1121 10:22:10.219202 20174 tablet_bootstrap.cc:439] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b: Bootstrap starting. I1121 10:22:10.219241 20175 tablet_bootstrap.cc:439] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa: No bootstrap required, opened a new log I1121 10:22:10.219271 20175 ts_tablet_manager.cc:1104] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:10.219408 20175 raft_consensus.cc:337] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.219420 20175 raft_consensus.cc:363] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:10.219427 20175 raft_consensus.cc:702] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Initialized, Role: FOLLOWER I1121 10:22:10.219483 20175 consensus_queue.cc:231] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.219552 20175 ts_tablet_manager.cc:1132] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:10.219588 20175 tablet_bootstrap.cc:439] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa: Bootstrap starting. I1121 10:22:10.219591 20174 tablet_bootstrap.cc:590] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b: No blocks or log segments found. Creating new log. I1121 10:22:10.219869 20175 tablet_bootstrap.cc:590] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa: No blocks or log segments found. Creating new log. I1121 10:22:10.220099 20174 tablet_bootstrap.cc:439] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b: No bootstrap required, opened a new log I1121 10:22:10.220134 20174 ts_tablet_manager.cc:1104] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:10.220268 20174 raft_consensus.cc:337] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.220283 20174 raft_consensus.cc:363] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:10.220289 20174 raft_consensus.cc:702] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Initialized, Role: FOLLOWER I1121 10:22:10.220342 20174 consensus_queue.cc:231] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.220422 20174 ts_tablet_manager.cc:1132] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:10.220440 20175 tablet_bootstrap.cc:439] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa: No bootstrap required, opened a new log I1121 10:22:10.220456 20174 tablet_bootstrap.cc:439] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b: Bootstrap starting. I1121 10:22:10.220469 20175 ts_tablet_manager.cc:1104] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:10.220659 20175 raft_consensus.cc:337] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.220679 20175 raft_consensus.cc:363] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:10.220697 20175 raft_consensus.cc:702] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Initialized, Role: FOLLOWER I1121 10:22:10.220749 20175 consensus_queue.cc:231] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.220825 20175 ts_tablet_manager.cc:1132] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:10.220860 20175 tablet_bootstrap.cc:439] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa: Bootstrap starting. I1121 10:22:10.221020 20174 tablet_bootstrap.cc:590] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b: No blocks or log segments found. Creating new log. I1121 10:22:10.221164 20175 tablet_bootstrap.cc:590] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa: No blocks or log segments found. Creating new log. I1121 10:22:10.221576 20174 tablet_bootstrap.cc:439] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b: No bootstrap required, opened a new log I1121 10:22:10.221614 20174 ts_tablet_manager.cc:1104] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:10.221698 20175 tablet_bootstrap.cc:439] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa: No bootstrap required, opened a new log I1121 10:22:10.221741 20175 ts_tablet_manager.cc:1104] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:10.221777 20174 raft_consensus.cc:337] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.221792 20174 raft_consensus.cc:363] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:10.221799 20174 raft_consensus.cc:702] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Initialized, Role: FOLLOWER I1121 10:22:10.221861 20174 consensus_queue.cc:231] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.221870 20175 raft_consensus.cc:337] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.221881 20175 raft_consensus.cc:363] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:10.221889 20175 raft_consensus.cc:702] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Initialized, Role: FOLLOWER I1121 10:22:10.221958 20174 ts_tablet_manager.cc:1132] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:10.221972 20175 consensus_queue.cc:231] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.222009 20174 tablet_bootstrap.cc:439] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b: Bootstrap starting. I1121 10:22:10.222045 20175 ts_tablet_manager.cc:1132] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:10.222324 20174 tablet_bootstrap.cc:590] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b: No blocks or log segments found. Creating new log. I1121 10:22:10.222877 20174 tablet_bootstrap.cc:439] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b: No bootstrap required, opened a new log I1121 10:22:10.222910 20174 ts_tablet_manager.cc:1104] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s I1121 10:22:10.223067 20174 raft_consensus.cc:337] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.223080 20174 raft_consensus.cc:363] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I1121 10:22:10.223091 20174 raft_consensus.cc:702] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Initialized, Role: FOLLOWER I1121 10:22:10.223135 20174 consensus_queue.cc:231] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.223212 20174 ts_tablet_manager.cc:1132] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:10.270257 20127 raft_consensus.cc:469] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:10.270371 20127 raft_consensus.cc:491] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.270460 20127 leader_election.cc:296] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:10.270789 20062 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5ab68244024a4864b50a76748a5a6c22" candidate_uuid: "8754ab0c37934104af1213092861d2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a73706e30e7445a93af7bd060d6103b" is_pre_election: true I1121 10:22:10.270833 20062 raft_consensus.cc:2359] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8754ab0c37934104af1213092861d2fa in term 0. I1121 10:22:10.270982 19618 leader_election.cc:310] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.271062 20127 raft_consensus.cc:2653] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:22:10.271083 20127 raft_consensus.cc:469] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:22:10.271096 20127 raft_consensus.cc:2905] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.271808 20127 raft_consensus.cc:491] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.271863 20127 leader_election.cc:296] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 election: Requested vote from peers 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:10.272050 20062 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "5ab68244024a4864b50a76748a5a6c22" candidate_uuid: "8754ab0c37934104af1213092861d2fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a73706e30e7445a93af7bd060d6103b" I1121 10:22:10.272075 20062 raft_consensus.cc:2905] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.272601 20062 raft_consensus.cc:2359] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8754ab0c37934104af1213092861d2fa in term 1. I1121 10:22:10.272766 19618 leader_election.cc:310] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.272825 20127 raft_consensus.cc:2653] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:10.272862 20127 raft_consensus.cc:664] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 1 LEADER]: Becoming Leader. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Running, Role: LEADER I1121 10:22:10.272917 20127 consensus_queue.cc:208] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } I1121 10:22:10.273593 19560 catalog_manager.cc:4143] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa reported cstate change: term changed from 0 to 1, leader changed from to 8754ab0c37934104af1213092861d2fa (127.5.70.1). New cstate: current_term: 1 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: UNKNOWN } } } I1121 10:22:10.302175 20134 raft_consensus.cc:469] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:10.302280 20134 raft_consensus.cc:491] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.302350 20134 leader_election.cc:296] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:10.302580 19679 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8f0b8462a15040e88aebbbf5d5bf9dcd" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" is_pre_election: true I1121 10:22:10.302624 19679 raft_consensus.cc:2359] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 0. I1121 10:22:10.302781 19995 leader_election.cc:310] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.302855 20134 raft_consensus.cc:2653] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:22:10.302881 20134 raft_consensus.cc:469] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:22:10.302889 20134 raft_consensus.cc:2905] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.303597 20134 raft_consensus.cc:491] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.303689 20134 leader_election.cc:296] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Requested vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:10.303922 19679 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "8f0b8462a15040e88aebbbf5d5bf9dcd" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" I1121 10:22:10.303952 19679 raft_consensus.cc:2905] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.304467 19679 raft_consensus.cc:2359] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 1. I1121 10:22:10.304653 19995 leader_election.cc:310] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.304738 20134 raft_consensus.cc:2653] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:10.304780 20134 raft_consensus.cc:664] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Becoming Leader. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Running, Role: LEADER I1121 10:22:10.304836 20134 consensus_queue.cc:208] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.305425 19560 catalog_manager.cc:4143] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b reported cstate change: term changed from 0 to 1, leader changed from to 4a73706e30e7445a93af7bd060d6103b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: UNKNOWN } } } I1121 10:22:10.407649 20134 raft_consensus.cc:469] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I1121 10:22:10.407761 20134 raft_consensus.cc:491] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.407838 20134 leader_election.cc:296] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:10.408187 19679 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "99f1a040116d4b36b79b74919f086440" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" is_pre_election: true I1121 10:22:10.408238 19679 raft_consensus.cc:2359] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 0. I1121 10:22:10.408414 19995 leader_election.cc:310] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.408561 20134 raft_consensus.cc:2653] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Leader pre-election won for term 1 I1121 10:22:10.408588 20134 raft_consensus.cc:469] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I1121 10:22:10.408598 20134 raft_consensus.cc:2905] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.409260 20134 raft_consensus.cc:491] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.409327 20134 leader_election.cc:296] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Requested vote from peers 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:10.409591 19679 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "99f1a040116d4b36b79b74919f086440" candidate_uuid: "4a73706e30e7445a93af7bd060d6103b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8754ab0c37934104af1213092861d2fa" I1121 10:22:10.409633 19679 raft_consensus.cc:2905] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa [term 0 FOLLOWER]: Advancing to term 1 I1121 10:22:10.410142 19679 raft_consensus.cc:2359] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a73706e30e7445a93af7bd060d6103b in term 1. I1121 10:22:10.410296 19995 leader_election.cc:310] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:10.410372 20134 raft_consensus.cc:2653] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Leader election won for term 1 I1121 10:22:10.410408 20134 raft_consensus.cc:664] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Becoming Leader. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Running, Role: LEADER I1121 10:22:10.410455 20134 consensus_queue.cc:208] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } I1121 10:22:10.411031 19560 catalog_manager.cc:4143] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: term changed from 0 to 1, leader changed from to 4a73706e30e7445a93af7bd060d6103b (127.5.70.4). New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: UNKNOWN } } } W1121 10:22:10.506516 20212 flags.cc:404] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_2' I1121 10:22:10.535398 20062 raft_consensus.cc:1181] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:22:10.535586 19679 raft_consensus.cc:1181] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch) I1121 10:22:10.535609 19681 raft_consensus.cc:1181] T 99f1a040116d4b36b79b74919f086440 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I1121 10:22:10.535658 20127 consensus_queue.cc:984] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:22:10.535856 20128 consensus_queue.cc:984] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I1121 10:22:10.535902 20165 consensus_queue.cc:984] T 99f1a040116d4b36b79b74919f086440 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 20.7552 ms time per row: 0.0103776 ms I1121 10:22:10.546020 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.70.2:36033 --local_ip_for_outbound_sockets=127.5.70.2 --webserver_port=45307 --webserver_interface=127.5.70.2 --tserver_master_addrs=127.5.70.62:46063 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34015 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:10.582273 20229 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:10.582427 20229 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:10.583190 20229 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:10.583210 20229 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:10.583243 20229 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.2 I1121 10:22:10.584422 20229 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34015 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.2:36033 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.70.2 --webserver_port=45307 --tserver_master_addrs=127.5.70.62:46063 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.20229 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.2 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:10.586239 20229 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:10.587373 20229 hybrid_clock.cc:183] HybridClock initialized: now 1574331730587350 us; error 29 us; skew 500 ppm I1121 10:22:10.587517 20229 fs_manager.cc:261] Metadata directory not provided I1121 10:22:10.587574 20229 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:10.588675 20229 fs_manager.cc:404] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:22:10.588887 20229 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:10.589011 20229 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:10.589761 20229 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:10.590189 20229 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal uuid: "389dbaf174a9403dbe2a244219e0dbb2" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:10.590281 20229 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:10.613986 20229 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:10.614385 20229 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:10.614403 20229 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:10.620014 20229 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.2:36033 I1121 10:22:10.620400 20229 webserver.cc:382] Webserver started at http://127.5.70.2:45307/ using document root and password file I1121 10:22:10.620748 20229 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-1/data/info.pb I1121 10:22:10.625102 20350 heartbeater.cc:322] Connected to a master server at 127.5.70.62:46063 I1121 10:22:10.625150 20350 heartbeater.cc:406] Registering TS with master... I1121 10:22:10.625262 20350 heartbeater.cc:455] Master 127.5.70.62:46063 requested a full tablet report, sending... I1121 10:22:10.625603 19560 ts_manager.cc:192] Re-registered known tserver with Master: 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2:36033) I1121 10:22:10.626399 19560 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.2:54203 I1121 10:22:10.629766 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 20229 I1121 10:22:10.629967 5400 external_mini_cluster.cc:1039] Running /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.70.3:36993 --local_ip_for_outbound_sockets=127.5.70.3 --webserver_port=44583 --webserver_interface=127.5.70.3 --tserver_master_addrs=127.5.70.62:46063 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --builtin_ntp_servers=127.5.70.20:34015 --builtin_ntp_poll_interval_ms=100 --ntp_initial_sync_wait_secs=10 --time_source=builtin --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W1121 10:22:10.666479 20353 flags.cc:404] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W1121 10:22:10.666646 20353 flags.cc:404] Enabled unsafe flag: --never_fsync=true W1121 10:22:10.667474 20353 flags.cc:404] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W1121 10:22:10.667495 20353 flags.cc:404] Enabled experimental flag: --ipki_server_key_size=1024 W1121 10:22:10.667508 20353 flags.cc:404] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.70.3 I1121 10:22:10.668702 20353 tablet_server_main.cc:78] Tablet server non-default flags: --builtin_ntp_poll_interval_ms=100 --builtin_ntp_servers=127.5.70.20:34015 --ntp_initial_sync_wait_secs=10 --time_source=builtin --fs_data_dirs=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.70.3:36993 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.70.3 --webserver_port=44583 --tserver_master_addrs=127.5.70.62:46063 --never_fsync=true --heap_profile_path=/tmp/kudu-tserver.20353 --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.70.3 --log_dir=/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.12.0-SNAPSHOT revision 0eb0a60f23251afbbe987e9bf88036b0b95f97ab build type RELEASE built by jenkins at 21 Nov 2019 01:42:44 PST on quasar-layfoo-1.vpc.cloudera.com build id 531882 I1121 10:22:10.670392 20353 hybrid_clock.cc:165] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize I1121 10:22:10.671525 20353 hybrid_clock.cc:183] HybridClock initialized: now 1574331730671479 us; error 57 us; skew 500 ppm I1121 10:22:10.671655 20353 fs_manager.cc:261] Metadata directory not provided I1121 10:22:10.671700 20353 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I1121 10:22:10.672843 20353 fs_manager.cc:404] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s I1121 10:22:10.673058 20353 env_posix.cc:1676] Not raising this process' open files per process limit of 1048576; it is already as high as it can go I1121 10:22:10.673190 20353 file_cache.cc:465] Constructed file cache lbm with capacity 419430 I1121 10:22:10.673913 20353 fs_manager.cc:424] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s I1121 10:22:10.674338 20353 fs_manager.cc:441] Opened local filesystem: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal uuid: "9a5dc7ff276144dfb28b69a0dd92218d" format_stamp: "Formatted at 2019-11-21 10:22:05 on dist-test-slave-dist-test-slave-5k2q" I1121 10:22:10.674396 20353 fs_report.cc:352] FS layout report -------------------- wal directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal metadata directory: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I1121 10:22:10.689422 20353 env_posix.cc:1676] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go I1121 10:22:10.689772 20353 ts_tablet_manager.cc:396] Loaded tablet metadata (0 total tablets, 0 live tablets) I1121 10:22:10.689792 20353 ts_tablet_manager.cc:416] Registered 0 tablets I1121 10:22:10.694873 20353 rpc_server.cc:211] RPC server started. Bound to: 127.5.70.3:36993 I1121 10:22:10.695222 20353 webserver.cc:382] Webserver started at http://127.5.70.3:44583/ using document root and password file I1121 10:22:10.695470 20353 server_base.cc:633] Dumped server information to /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-2/data/info.pb I1121 10:22:10.699681 20474 heartbeater.cc:322] Connected to a master server at 127.5.70.62:46063 I1121 10:22:10.699726 20474 heartbeater.cc:406] Registering TS with master... I1121 10:22:10.699811 20474 heartbeater.cc:455] Master 127.5.70.62:46063 requested a full tablet report, sending... I1121 10:22:10.700069 19560 ts_manager.cc:192] Re-registered known tserver with Master: 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3:36993) I1121 10:22:10.700649 19560 master_service.cc:351] Signed X509 certificate for tserver {username='slave'} at 127.5.70.3:53951 I1121 10:22:10.703934 5400 external_mini_cluster.cc:1101] Started /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver as pid 20353 I1121 10:22:10.884470 19679 consensus_queue.cc:208] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } I1121 10:22:10.885116 20062 raft_consensus.cc:1181] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:22:10.885414 20157 consensus_queue.cc:984] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:22:10.886138 20127 raft_consensus.cc:2801] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } } I1121 10:22:10.886287 20062 raft_consensus.cc:2801] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } } I1121 10:22:10.887799 19562 catalog_manager.cc:4143] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: config changed from index -1 to 4, NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) added. New cstate: current_term: 1 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } } W1121 10:22:10.888388 19617 consensus_peers.cc:473] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa -> Peer 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2:36033): Couldn't send request to peer 389dbaf174a9403dbe2a244219e0dbb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ab68244024a4864b50a76748a5a6c22. This is attempt 1: this message will repeat every 5th retry. I1121 10:22:10.892383 20062 consensus_queue.cc:208] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } I1121 10:22:10.892833 19679 raft_consensus.cc:1181] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:22:10.893049 20165 consensus_queue.cc:984] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:22:10.894031 20167 raft_consensus.cc:2801] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } } I1121 10:22:10.894212 19679 raft_consensus.cc:2801] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } } I1121 10:22:10.895395 19555 catalog_manager.cc:4143] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa reported cstate change: config changed from index -1 to 4, NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) added. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } } W1121 10:22:10.895877 19993 consensus_peers.cc:473] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b -> Peer 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3:36993): Couldn't send request to peer 9a5dc7ff276144dfb28b69a0dd92218d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d267eb692ae4744a5d2d38265f30121. This is attempt 1: this message will repeat every 5th retry. I1121 10:22:10.900262 19679 consensus_queue.cc:208] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } I1121 10:22:10.900756 20062 raft_consensus.cc:1181] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:22:10.900857 19617 consensus_peers.cc:473] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa -> Peer 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2:36033): Couldn't send request to peer 389dbaf174a9403dbe2a244219e0dbb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81897d1f2f904d2e89095a4818043edf. This is attempt 1: this message will repeat every 5th retry. I1121 10:22:10.901021 20157 consensus_queue.cc:984] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:22:10.901741 20156 raft_consensus.cc:2801] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } } I1121 10:22:10.901911 20062 raft_consensus.cc:2801] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } } I1121 10:22:10.902868 19562 catalog_manager.cc:4143] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b reported cstate change: config changed from index -1 to 4, NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) added. New cstate: current_term: 1 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } } I1121 10:22:10.907447 20062 consensus_queue.cc:208] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } I1121 10:22:10.907843 19679 raft_consensus.cc:1181] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:22:10.908041 19993 consensus_peers.cc:473] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b -> Peer 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3:36993): Couldn't send request to peer 9a5dc7ff276144dfb28b69a0dd92218d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f0b8462a15040e88aebbbf5d5bf9dcd. This is attempt 1: this message will repeat every 5th retry. I1121 10:22:10.908090 20165 consensus_queue.cc:984] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:22:10.908933 20128 raft_consensus.cc:2801] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } } I1121 10:22:10.909075 19679 raft_consensus.cc:2801] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } } I1121 10:22:10.910153 19562 catalog_manager.cc:4143] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa reported cstate change: config changed from index -1 to 4, NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) added. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } } I1121 10:22:10.914376 20062 consensus_queue.cc:208] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } I1121 10:22:10.914752 19679 raft_consensus.cc:1181] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) W1121 10:22:10.914881 19993 consensus_peers.cc:473] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b -> Peer 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3:36993): Couldn't send request to peer 9a5dc7ff276144dfb28b69a0dd92218d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6bcafd84c0949b6b7d7fa7dc447f620. This is attempt 1: this message will repeat every 5th retry. I1121 10:22:10.915072 20165 consensus_queue.cc:984] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:22:10.915664 20167 raft_consensus.cc:2801] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } } I1121 10:22:10.915814 19679 raft_consensus.cc:2801] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } } I1121 10:22:10.916684 19555 catalog_manager.cc:4143] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: config changed from index -1 to 4, NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) added. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I1121 10:22:10.921566 20062 consensus_queue.cc:208] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } I1121 10:22:10.922010 19679 raft_consensus.cc:1181] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch) I1121 10:22:10.922230 20134 consensus_queue.cc:984] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I1121 10:22:10.922901 20128 raft_consensus.cc:2801] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } } I1121 10:22:10.923107 19679 raft_consensus.cc:2801] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } } I1121 10:22:10.923854 19562 catalog_manager.cc:4143] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b reported cstate change: config changed from index -1 to 4, NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) added. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W1121 10:22:10.925076 19993 consensus_peers.cc:473] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b -> Peer 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2:36033): Couldn't send request to peer 389dbaf174a9403dbe2a244219e0dbb2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74d663e39077406590cb0c9d4204ac1d. This is attempt 1: this message will repeat every 5th retry. I1121 10:22:11.312501 20698 ts_tablet_manager.cc:721] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: Initiating tablet copy from peer 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:11.312501 20697 ts_tablet_manager.cc:721] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: Initiating tablet copy from peer 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:11.312590 20697 tablet_copy_client.cc:243] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:44823 I1121 10:22:11.312594 20698 tablet_copy_client.cc:243] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:44823 I1121 10:22:11.315235 20082 tablet_copy_service.cc:135] P 4a73706e30e7445a93af7bd060d6103b: Received BeginTabletCopySession request for tablet 8f0b8462a15040e88aebbbf5d5bf9dcd from peer 9a5dc7ff276144dfb28b69a0dd92218d ({username='slave'} at 127.5.70.3:53895) I1121 10:22:11.315277 20082 tablet_copy_service.cc:156] P 4a73706e30e7445a93af7bd060d6103b: Beginning new tablet copy session on tablet 8f0b8462a15040e88aebbbf5d5bf9dcd from peer 9a5dc7ff276144dfb28b69a0dd92218d at {username='slave'} at 127.5.70.3:53895: session id = 9a5dc7ff276144dfb28b69a0dd92218d-8f0b8462a15040e88aebbbf5d5bf9dcd I1121 10:22:11.315310 20081 tablet_copy_service.cc:135] P 4a73706e30e7445a93af7bd060d6103b: Received BeginTabletCopySession request for tablet 74d663e39077406590cb0c9d4204ac1d from peer 389dbaf174a9403dbe2a244219e0dbb2 ({username='slave'} at 127.5.70.2:53433) I1121 10:22:11.315337 20081 tablet_copy_service.cc:156] P 4a73706e30e7445a93af7bd060d6103b: Beginning new tablet copy session on tablet 74d663e39077406590cb0c9d4204ac1d from peer 389dbaf174a9403dbe2a244219e0dbb2 at {username='slave'} at 127.5.70.2:53433: session id = 389dbaf174a9403dbe2a244219e0dbb2-74d663e39077406590cb0c9d4204ac1d I1121 10:22:11.315948 20081 tablet_copy_source_session.cc:204] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:22:11.315948 20082 tablet_copy_source_session.cc:204] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:22:11.316808 20697 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 74d663e39077406590cb0c9d4204ac1d. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:11.316808 20698 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 8f0b8462a15040e88aebbbf5d5bf9dcd. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:11.319948 20697 tablet_copy_client.cc:581] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Starting download of 0 data blocks... I1121 10:22:11.319948 20698 tablet_copy_client.cc:581] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Starting download of 0 data blocks... I1121 10:22:11.320066 20697 tablet_copy_client.cc:544] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Starting download of 1 WAL segments... I1121 10:22:11.320066 20698 tablet_copy_client.cc:544] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Starting download of 1 WAL segments... I1121 10:22:11.320971 20697 tablet_copy_client.cc:420] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:22:11.320971 20698 tablet_copy_client.cc:420] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:22:11.321724 20697 tablet_bootstrap.cc:439] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: Bootstrap starting. I1121 10:22:11.321724 20698 tablet_bootstrap.cc:439] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: Bootstrap starting. I1121 10:22:11.323505 20697 log.cc:810] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: Log is configured to *not* fsync() on all Append() calls I1121 10:22:11.323511 20698 log.cc:810] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: Log is configured to *not* fsync() on all Append() calls I1121 10:22:11.328555 20697 tablet_bootstrap.cc:439] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:22:11.328555 20698 tablet_bootstrap.cc:439] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:22:11.328796 20697 tablet_bootstrap.cc:439] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: Bootstrap complete. I1121 10:22:11.328801 20698 tablet_bootstrap.cc:439] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: Bootstrap complete. I1121 10:22:11.328925 20698 ts_tablet_manager.cc:1104] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s I1121 10:22:11.328929 20697 ts_tablet_manager.cc:1104] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s I1121 10:22:11.330070 20698 raft_consensus.cc:337] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } I1121 10:22:11.330070 20697 raft_consensus.cc:337] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } I1121 10:22:11.330137 20697 raft_consensus.cc:702] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 389dbaf174a9403dbe2a244219e0dbb2, State: Initialized, Role: LEARNER I1121 10:22:11.330137 20698 raft_consensus.cc:702] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a5dc7ff276144dfb28b69a0dd92218d, State: Initialized, Role: LEARNER I1121 10:22:11.330231 20697 consensus_queue.cc:231] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } I1121 10:22:11.330231 20698 consensus_queue.cc:231] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } I1121 10:22:11.330442 20697 ts_tablet_manager.cc:1132] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s I1121 10:22:11.330440 20474 heartbeater.cc:447] Master 127.5.70.62:46063 was elected leader, sending a full tablet report... I1121 10:22:11.330442 20698 ts_tablet_manager.cc:1132] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s I1121 10:22:11.330608 20350 heartbeater.cc:447] Master 127.5.70.62:46063 was elected leader, sending a full tablet report... I1121 10:22:11.330718 20082 tablet_copy_service.cc:325] P 4a73706e30e7445a93af7bd060d6103b: Request end of tablet copy session 389dbaf174a9403dbe2a244219e0dbb2-74d663e39077406590cb0c9d4204ac1d received from {username='slave'} at 127.5.70.2:53433 I1121 10:22:11.330761 20082 tablet_copy_service.cc:413] P 4a73706e30e7445a93af7bd060d6103b: ending tablet copy session 389dbaf174a9403dbe2a244219e0dbb2-74d663e39077406590cb0c9d4204ac1d on tablet 74d663e39077406590cb0c9d4204ac1d with peer 389dbaf174a9403dbe2a244219e0dbb2 I1121 10:22:11.330857 20081 tablet_copy_service.cc:325] P 4a73706e30e7445a93af7bd060d6103b: Request end of tablet copy session 9a5dc7ff276144dfb28b69a0dd92218d-8f0b8462a15040e88aebbbf5d5bf9dcd received from {username='slave'} at 127.5.70.3:53895 I1121 10:22:11.330878 20081 tablet_copy_service.cc:413] P 4a73706e30e7445a93af7bd060d6103b: ending tablet copy session 9a5dc7ff276144dfb28b69a0dd92218d-8f0b8462a15040e88aebbbf5d5bf9dcd on tablet 8f0b8462a15040e88aebbbf5d5bf9dcd with peer 9a5dc7ff276144dfb28b69a0dd92218d I1121 10:22:11.362646 20697 ts_tablet_manager.cc:721] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: Initiating tablet copy from peer 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:11.362776 20697 tablet_copy_client.cc:243] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:33643 I1121 10:22:11.365893 19706 tablet_copy_service.cc:135] P 8754ab0c37934104af1213092861d2fa: Received BeginTabletCopySession request for tablet 5ab68244024a4864b50a76748a5a6c22 from peer 389dbaf174a9403dbe2a244219e0dbb2 ({username='slave'} at 127.5.70.2:45181) I1121 10:22:11.365933 19706 tablet_copy_service.cc:156] P 8754ab0c37934104af1213092861d2fa: Beginning new tablet copy session on tablet 5ab68244024a4864b50a76748a5a6c22 from peer 389dbaf174a9403dbe2a244219e0dbb2 at {username='slave'} at 127.5.70.2:45181: session id = 389dbaf174a9403dbe2a244219e0dbb2-5ab68244024a4864b50a76748a5a6c22 I1121 10:22:11.366550 20698 ts_tablet_manager.cc:721] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: Initiating tablet copy from peer 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:11.366593 20698 tablet_copy_client.cc:243] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:44823 I1121 10:22:11.366689 19706 tablet_copy_source_session.cc:204] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:22:11.366763 20081 tablet_copy_service.cc:135] P 4a73706e30e7445a93af7bd060d6103b: Received BeginTabletCopySession request for tablet 0d267eb692ae4744a5d2d38265f30121 from peer 9a5dc7ff276144dfb28b69a0dd92218d ({username='slave'} at 127.5.70.3:53895) I1121 10:22:11.366793 20081 tablet_copy_service.cc:156] P 4a73706e30e7445a93af7bd060d6103b: Beginning new tablet copy session on tablet 0d267eb692ae4744a5d2d38265f30121 from peer 9a5dc7ff276144dfb28b69a0dd92218d at {username='slave'} at 127.5.70.3:53895: session id = 9a5dc7ff276144dfb28b69a0dd92218d-0d267eb692ae4744a5d2d38265f30121 I1121 10:22:11.367141 20697 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 5ab68244024a4864b50a76748a5a6c22. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:11.367296 20081 tablet_copy_source_session.cc:204] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:22:11.367668 20698 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 0d267eb692ae4744a5d2d38265f30121. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:11.368387 20697 tablet_copy_client.cc:581] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Starting download of 0 data blocks... I1121 10:22:11.368490 20697 tablet_copy_client.cc:544] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Starting download of 1 WAL segments... I1121 10:22:11.368914 20698 tablet_copy_client.cc:581] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Starting download of 0 data blocks... I1121 10:22:11.369002 20698 tablet_copy_client.cc:544] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Starting download of 1 WAL segments... I1121 10:22:11.369300 20697 tablet_copy_client.cc:420] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:22:11.369534 20698 tablet_copy_client.cc:420] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:22:11.369905 20697 tablet_bootstrap.cc:439] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: Bootstrap starting. I1121 10:22:11.369963 20698 tablet_bootstrap.cc:439] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: Bootstrap starting. I1121 10:22:11.373508 20697 tablet_bootstrap.cc:439] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:22:11.373689 20697 tablet_bootstrap.cc:439] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: Bootstrap complete. I1121 10:22:11.373788 20697 ts_tablet_manager.cc:1104] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s I1121 10:22:11.373908 20698 tablet_bootstrap.cc:439] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=623 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:22:11.373932 20697 raft_consensus.cc:337] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } I1121 10:22:11.373950 20697 raft_consensus.cc:702] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 389dbaf174a9403dbe2a244219e0dbb2, State: Initialized, Role: LEARNER I1121 10:22:11.373997 20697 consensus_queue.cc:231] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } I1121 10:22:11.374109 20698 tablet_bootstrap.cc:439] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: Bootstrap complete. I1121 10:22:11.374208 20698 ts_tablet_manager.cc:1104] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I1121 10:22:11.374385 20698 raft_consensus.cc:337] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } I1121 10:22:11.374409 20698 raft_consensus.cc:702] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a5dc7ff276144dfb28b69a0dd92218d, State: Initialized, Role: LEARNER I1121 10:22:11.374460 20698 consensus_queue.cc:231] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } I1121 10:22:11.374747 20698 ts_tablet_manager.cc:1132] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s I1121 10:22:11.375051 20081 tablet_copy_service.cc:325] P 4a73706e30e7445a93af7bd060d6103b: Request end of tablet copy session 9a5dc7ff276144dfb28b69a0dd92218d-0d267eb692ae4744a5d2d38265f30121 received from {username='slave'} at 127.5.70.3:53895 I1121 10:22:11.375075 20081 tablet_copy_service.cc:413] P 4a73706e30e7445a93af7bd060d6103b: ending tablet copy session 9a5dc7ff276144dfb28b69a0dd92218d-0d267eb692ae4744a5d2d38265f30121 on tablet 0d267eb692ae4744a5d2d38265f30121 with peer 9a5dc7ff276144dfb28b69a0dd92218d I1121 10:22:11.375130 20697 ts_tablet_manager.cc:1132] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I1121 10:22:11.375349 19706 tablet_copy_service.cc:325] P 8754ab0c37934104af1213092861d2fa: Request end of tablet copy session 389dbaf174a9403dbe2a244219e0dbb2-5ab68244024a4864b50a76748a5a6c22 received from {username='slave'} at 127.5.70.2:45181 I1121 10:22:11.375377 19706 tablet_copy_service.cc:413] P 8754ab0c37934104af1213092861d2fa: ending tablet copy session 389dbaf174a9403dbe2a244219e0dbb2-5ab68244024a4864b50a76748a5a6c22 on tablet 5ab68244024a4864b50a76748a5a6c22 with peer 389dbaf174a9403dbe2a244219e0dbb2 I1121 10:22:11.378903 20698 ts_tablet_manager.cc:721] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: Initiating tablet copy from peer 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823) I1121 10:22:11.378942 20698 tablet_copy_client.cc:243] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.4:44823 I1121 10:22:11.379155 20081 tablet_copy_service.cc:135] P 4a73706e30e7445a93af7bd060d6103b: Received BeginTabletCopySession request for tablet b6bcafd84c0949b6b7d7fa7dc447f620 from peer 9a5dc7ff276144dfb28b69a0dd92218d ({username='slave'} at 127.5.70.3:53895) I1121 10:22:11.379194 20081 tablet_copy_service.cc:156] P 4a73706e30e7445a93af7bd060d6103b: Beginning new tablet copy session on tablet b6bcafd84c0949b6b7d7fa7dc447f620 from peer 9a5dc7ff276144dfb28b69a0dd92218d at {username='slave'} at 127.5.70.3:53895: session id = 9a5dc7ff276144dfb28b69a0dd92218d-b6bcafd84c0949b6b7d7fa7dc447f620 I1121 10:22:11.379561 20081 tablet_copy_source_session.cc:204] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:22:11.379910 20698 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet b6bcafd84c0949b6b7d7fa7dc447f620. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:11.380971 20698 tablet_copy_client.cc:581] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Starting download of 0 data blocks... I1121 10:22:11.381067 20698 tablet_copy_client.cc:544] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Starting download of 1 WAL segments... I1121 10:22:11.381619 20698 tablet_copy_client.cc:420] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:22:11.382256 20698 tablet_bootstrap.cc:439] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: Bootstrap starting. I1121 10:22:11.386229 20698 tablet_bootstrap.cc:439] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:22:11.386415 20698 tablet_bootstrap.cc:439] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: Bootstrap complete. I1121 10:22:11.386508 20698 ts_tablet_manager.cc:1104] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s I1121 10:22:11.386631 20698 raft_consensus.cc:337] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } I1121 10:22:11.386656 20698 raft_consensus.cc:702] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9a5dc7ff276144dfb28b69a0dd92218d, State: Initialized, Role: LEARNER I1121 10:22:11.386718 20698 consensus_queue.cc:231] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: NON_VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: true } } I1121 10:22:11.386889 20698 ts_tablet_manager.cc:1132] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d: Time spent starting tablet: real 0.000s user 0.002s sys 0.000s I1121 10:22:11.387061 20081 tablet_copy_service.cc:325] P 4a73706e30e7445a93af7bd060d6103b: Request end of tablet copy session 9a5dc7ff276144dfb28b69a0dd92218d-b6bcafd84c0949b6b7d7fa7dc447f620 received from {username='slave'} at 127.5.70.3:53895 I1121 10:22:11.387084 20081 tablet_copy_service.cc:413] P 4a73706e30e7445a93af7bd060d6103b: ending tablet copy session 9a5dc7ff276144dfb28b69a0dd92218d-b6bcafd84c0949b6b7d7fa7dc447f620 on tablet b6bcafd84c0949b6b7d7fa7dc447f620 with peer 9a5dc7ff276144dfb28b69a0dd92218d I1121 10:22:11.441557 20697 ts_tablet_manager.cc:721] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: Initiating tablet copy from peer 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643) I1121 10:22:11.441612 20697 tablet_copy_client.cc:243] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.70.1:33643 I1121 10:22:11.441871 19706 tablet_copy_service.cc:135] P 8754ab0c37934104af1213092861d2fa: Received BeginTabletCopySession request for tablet 81897d1f2f904d2e89095a4818043edf from peer 389dbaf174a9403dbe2a244219e0dbb2 ({username='slave'} at 127.5.70.2:45181) I1121 10:22:11.441908 19706 tablet_copy_service.cc:156] P 8754ab0c37934104af1213092861d2fa: Beginning new tablet copy session on tablet 81897d1f2f904d2e89095a4818043edf from peer 389dbaf174a9403dbe2a244219e0dbb2 at {username='slave'} at 127.5.70.2:45181: session id = 389dbaf174a9403dbe2a244219e0dbb2-81897d1f2f904d2e89095a4818043edf I1121 10:22:11.442389 19706 tablet_copy_source_session.cc:204] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa: Tablet Copy: opened 0 blocks and 1 log segments I1121 10:22:11.442737 20697 data_dirs.cc:946] Could only allocate 1 dirs of requested 3 for tablet 81897d1f2f904d2e89095a4818043edf. 1 dirs total, 0 dirs full, 0 dirs failed I1121 10:22:11.443560 20697 tablet_copy_client.cc:581] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Starting download of 0 data blocks... I1121 10:22:11.443682 20697 tablet_copy_client.cc:544] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Starting download of 1 WAL segments... I1121 10:22:11.444322 20697 tablet_copy_client.cc:420] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: tablet copy: Tablet Copy complete. Replacing tablet superblock. I1121 10:22:11.444844 20697 tablet_bootstrap.cc:439] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: Bootstrap starting. I1121 10:22:11.447721 20697 tablet_bootstrap.cc:439] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I1121 10:22:11.447849 20697 tablet_bootstrap.cc:439] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: Bootstrap complete. I1121 10:22:11.447916 20697 ts_tablet_manager.cc:1104] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I1121 10:22:11.448031 20697 raft_consensus.cc:337] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } I1121 10:22:11.448050 20697 raft_consensus.cc:702] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 389dbaf174a9403dbe2a244219e0dbb2, State: Initialized, Role: LEARNER I1121 10:22:11.448081 20697 consensus_queue.cc:231] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: NON_VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: true } } I1121 10:22:11.448143 20697 ts_tablet_manager.cc:1132] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s I1121 10:22:11.448405 19706 tablet_copy_service.cc:325] P 8754ab0c37934104af1213092861d2fa: Request end of tablet copy session 389dbaf174a9403dbe2a244219e0dbb2-81897d1f2f904d2e89095a4818043edf received from {username='slave'} at 127.5.70.2:45181 I1121 10:22:11.448426 19706 tablet_copy_service.cc:413] P 8754ab0c37934104af1213092861d2fa: ending tablet copy session 389dbaf174a9403dbe2a244219e0dbb2-81897d1f2f904d2e89095a4818043edf on tablet 81897d1f2f904d2e89095a4818043edf with peer 389dbaf174a9403dbe2a244219e0dbb2 I1121 10:22:11.766554 20430 raft_consensus.cc:1123] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:22:11.835713 20431 raft_consensus.cc:1123] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:22:11.835767 20430 raft_consensus.cc:1123] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:22:11.884759 20305 raft_consensus.cc:1123] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:22:11.894928 20305 raft_consensus.cc:1123] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:22:11.934360 20305 raft_consensus.cc:1123] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I1121 10:22:12.183328 20802 raft_consensus.cc:971] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b: attempting to promote NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d to VOTER I1121 10:22:12.183548 20802 consensus_queue.cc:208] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } I1121 10:22:12.184031 20430 raft_consensus.cc:1181] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.184121 19679 raft_consensus.cc:1181] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.184371 20134 consensus_queue.cc:984] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.184468 20165 consensus_queue.cc:984] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.185621 20801 raft_consensus.cc:2801] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.185750 19679 raft_consensus.cc:2801] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.185838 20430 raft_consensus.cc:2801] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.187018 19562 catalog_manager.cc:4143] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:22:12.188138 20062 consensus_queue.cc:208] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } I1121 10:22:12.188426 20430 raft_consensus.cc:1181] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:22:12.188827 20134 consensus_queue.cc:984] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:22:12.189502 20165 raft_consensus.cc:2801] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8754ab0c37934104af1213092861d2fa (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.189671 20430 raft_consensus.cc:2801] T 0d267eb692ae4744a5d2d38265f30121 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8754ab0c37934104af1213092861d2fa (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.190407 19546 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 0d267eb692ae4744a5d2d38265f30121 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:22:12.190537 19555 catalog_manager.cc:4143] T 0d267eb692ae4744a5d2d38265f30121 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: config changed from index 5 to 6, VOTER 8754ab0c37934104af1213092861d2fa (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:22:12.191267 19650 tablet_service.cc:1067] Processing DeleteTablet for tablet 0d267eb692ae4744a5d2d38265f30121 with delete_type TABLET_DATA_TOMBSTONED (TS 8754ab0c37934104af1213092861d2fa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33726 I1121 10:22:12.191422 20842 tablet_replica.cc:282] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa: stopping tablet replica I1121 10:22:12.191512 20842 raft_consensus.cc:2156] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:22:12.191555 20842 raft_consensus.cc:2183] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:22:12.192004 20842 ts_tablet_manager.cc:1457] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:22:12.193565 20842 ts_tablet_manager.cc:1470] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:22:12.193603 20842 log.cc:1153] T 0d267eb692ae4744a5d2d38265f30121 P 8754ab0c37934104af1213092861d2fa: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/0d267eb692ae4744a5d2d38265f30121 I1121 10:22:12.194023 19547 catalog_manager.cc:3512] TS 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643): tablet 0d267eb692ae4744a5d2d38265f30121 (table rebalance_test_table_1 [id=7cd596ea2e5c4cdb9985bbd0788e6f8c]) successfully deleted I1121 10:22:12.293465 20804 raft_consensus.cc:971] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa: attempting to promote NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 to VOTER I1121 10:22:12.293685 20804 consensus_queue.cc:208] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } I1121 10:22:12.294220 20305 raft_consensus.cc:1181] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.294312 20062 raft_consensus.cc:1181] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.294656 20804 consensus_queue.cc:984] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.294778 20157 consensus_queue.cc:984] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.295388 20804 raft_consensus.cc:2801] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.295539 20062 raft_consensus.cc:2801] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.296602 19555 catalog_manager.cc:4143] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b reported cstate change: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.297005 20305 raft_consensus.cc:2801] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.298386 19679 consensus_queue.cc:208] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } I1121 10:22:12.298799 20305 raft_consensus.cc:1181] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:22:12.299096 20804 consensus_queue.cc:984] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:22:12.299904 20899 raft_consensus.cc:2801] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.300048 20305 raft_consensus.cc:2801] T 81897d1f2f904d2e89095a4818043edf P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.300738 19547 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 81897d1f2f904d2e89095a4818043edf with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:22:12.300884 19555 catalog_manager.cc:4143] T 81897d1f2f904d2e89095a4818043edf P 8754ab0c37934104af1213092861d2fa reported cstate change: config changed from index 5 to 6, VOTER 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:22:12.301625 20041 tablet_service.cc:1067] Processing DeleteTablet for tablet 81897d1f2f904d2e89095a4818043edf with delete_type TABLET_DATA_TOMBSTONED (TS 4a73706e30e7445a93af7bd060d6103b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59982 I1121 10:22:12.301776 20902 tablet_replica.cc:282] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b: stopping tablet replica I1121 10:22:12.301846 20902 raft_consensus.cc:2156] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:22:12.301882 20902 raft_consensus.cc:2183] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:22:12.302356 20902 ts_tablet_manager.cc:1457] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:22:12.303833 20902 ts_tablet_manager.cc:1470] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:22:12.303927 20902 log.cc:1153] T 81897d1f2f904d2e89095a4818043edf P 4a73706e30e7445a93af7bd060d6103b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/81897d1f2f904d2e89095a4818043edf I1121 10:22:12.304342 19546 catalog_manager.cc:3512] TS 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823): tablet 81897d1f2f904d2e89095a4818043edf (table rebalance_test_table_1 [id=7cd596ea2e5c4cdb9985bbd0788e6f8c]) successfully deleted I1121 10:22:12.315209 20165 raft_consensus.cc:971] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b: attempting to promote NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 to VOTER I1121 10:22:12.315402 20165 consensus_queue.cc:208] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } I1121 10:22:12.315871 20305 raft_consensus.cc:1181] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.315963 19679 raft_consensus.cc:1181] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.316110 20134 consensus_queue.cc:984] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.316181 20165 consensus_queue.cc:984] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.317322 20165 raft_consensus.cc:2801] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.317448 20305 raft_consensus.cc:2801] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.317543 19679 raft_consensus.cc:2801] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.318715 19562 catalog_manager.cc:4143] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa reported cstate change: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.319844 20062 consensus_queue.cc:208] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } I1121 10:22:12.320127 20305 raft_consensus.cc:1181] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:22:12.320343 20134 consensus_queue.cc:984] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:22:12.320931 20802 raft_consensus.cc:2801] T 74d663e39077406590cb0c9d4204ac1d P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8754ab0c37934104af1213092861d2fa (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.321091 20305 raft_consensus.cc:2801] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8754ab0c37934104af1213092861d2fa (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.321698 19546 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 74d663e39077406590cb0c9d4204ac1d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:22:12.321900 19560 catalog_manager.cc:4143] T 74d663e39077406590cb0c9d4204ac1d P 389dbaf174a9403dbe2a244219e0dbb2 reported cstate change: config changed from index 5 to 6, VOTER 8754ab0c37934104af1213092861d2fa (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.322574 19650 tablet_service.cc:1067] Processing DeleteTablet for tablet 74d663e39077406590cb0c9d4204ac1d with delete_type TABLET_DATA_TOMBSTONED (TS 8754ab0c37934104af1213092861d2fa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33726 I1121 10:22:12.322662 20842 tablet_replica.cc:282] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: stopping tablet replica I1121 10:22:12.322728 20842 raft_consensus.cc:2156] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:22:12.322751 20842 raft_consensus.cc:2183] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:22:12.323145 20842 ts_tablet_manager.cc:1457] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:22:12.324466 20842 ts_tablet_manager.cc:1470] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:22:12.324501 20842 log.cc:1153] T 74d663e39077406590cb0c9d4204ac1d P 8754ab0c37934104af1213092861d2fa: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/74d663e39077406590cb0c9d4204ac1d I1121 10:22:12.324929 19547 catalog_manager.cc:3512] TS 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643): tablet 74d663e39077406590cb0c9d4204ac1d (table rebalance_test_table_0 [id=dca22bab1f104febb0f86c11e01c69b5]) successfully deleted I1121 10:22:12.386523 20804 raft_consensus.cc:971] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa: attempting to promote NON_VOTER 389dbaf174a9403dbe2a244219e0dbb2 to VOTER I1121 10:22:12.386755 20804 consensus_queue.cc:208] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } I1121 10:22:12.387254 20305 raft_consensus.cc:1181] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 LEARNER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.387291 20062 raft_consensus.cc:1181] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.387466 20899 consensus_queue.cc:984] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.387559 20804 consensus_queue.cc:984] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.388348 20804 raft_consensus.cc:2801] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.388502 20062 raft_consensus.cc:2801] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.388576 20305 raft_consensus.cc:2801] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.389598 19555 catalog_manager.cc:4143] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b reported cstate change: config changed from index 4 to 5, 389dbaf174a9403dbe2a244219e0dbb2 (127.5.70.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.390662 19679 consensus_queue.cc:208] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } I1121 10:22:12.390961 20305 raft_consensus.cc:1181] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:22:12.391176 20899 consensus_queue.cc:984] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:22:12.391803 20157 raft_consensus.cc:2801] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.391949 20305 raft_consensus.cc:2801] T 5ab68244024a4864b50a76748a5a6c22 P 389dbaf174a9403dbe2a244219e0dbb2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } } } I1121 10:22:12.392796 19547 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 5ab68244024a4864b50a76748a5a6c22 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:22:12.392822 19560 catalog_manager.cc:4143] T 5ab68244024a4864b50a76748a5a6c22 P 8754ab0c37934104af1213092861d2fa reported cstate change: config changed from index 5 to 6, VOTER 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) evicted. New cstate: current_term: 1 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "389dbaf174a9403dbe2a244219e0dbb2" member_type: VOTER last_known_addr { host: "127.5.70.2" port: 36033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:22:12.393458 20041 tablet_service.cc:1067] Processing DeleteTablet for tablet 5ab68244024a4864b50a76748a5a6c22 with delete_type TABLET_DATA_TOMBSTONED (TS 4a73706e30e7445a93af7bd060d6103b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59982 I1121 10:22:12.393540 20902 tablet_replica.cc:282] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b: stopping tablet replica I1121 10:22:12.393602 20902 raft_consensus.cc:2156] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:22:12.393628 20902 raft_consensus.cc:2183] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:22:12.393932 20902 ts_tablet_manager.cc:1457] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:22:12.395232 20902 ts_tablet_manager.cc:1470] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:22:12.395268 20902 log.cc:1153] T 5ab68244024a4864b50a76748a5a6c22 P 4a73706e30e7445a93af7bd060d6103b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/5ab68244024a4864b50a76748a5a6c22 I1121 10:22:12.395542 19546 catalog_manager.cc:3512] TS 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823): tablet 5ab68244024a4864b50a76748a5a6c22 (table rebalance_test_table_2 [id=302bc66d906a41448959053fb1bde011]) successfully deleted I1121 10:22:12.396421 20134 raft_consensus.cc:971] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b: attempting to promote NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d to VOTER I1121 10:22:12.396574 20134 consensus_queue.cc:208] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } I1121 10:22:12.396962 19679 raft_consensus.cc:1181] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.396966 20430 raft_consensus.cc:1181] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.397166 20801 consensus_queue.cc:984] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.397328 20134 consensus_queue.cc:984] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.398341 20165 raft_consensus.cc:2801] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.398470 19679 raft_consensus.cc:2801] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.398471 20430 raft_consensus.cc:2801] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.399422 19562 catalog_manager.cc:4143] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d reported cstate change: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.400282 20062 consensus_queue.cc:208] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } I1121 10:22:12.400599 20430 raft_consensus.cc:1181] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch) I1121 10:22:12.400849 20802 consensus_queue.cc:984] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:22:12.401535 20134 raft_consensus.cc:2801] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8754ab0c37934104af1213092861d2fa (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.401727 20430 raft_consensus.cc:2801] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8754ab0c37934104af1213092861d2fa (127.5.70.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.402217 19546 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet 8f0b8462a15040e88aebbbf5d5bf9dcd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:22:12.402529 19560 catalog_manager.cc:4143] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 9a5dc7ff276144dfb28b69a0dd92218d reported cstate change: config changed from index 5 to 6, VOTER 8754ab0c37934104af1213092861d2fa (127.5.70.1) evicted. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.403316 19650 tablet_service.cc:1067] Processing DeleteTablet for tablet 8f0b8462a15040e88aebbbf5d5bf9dcd with delete_type TABLET_DATA_TOMBSTONED (TS 8754ab0c37934104af1213092861d2fa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33726 I1121 10:22:12.403389 20842 tablet_replica.cc:282] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa: stopping tablet replica I1121 10:22:12.403436 20842 raft_consensus.cc:2156] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Raft consensus shutting down. I1121 10:22:12.403460 20842 raft_consensus.cc:2183] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Raft consensus is shut down! I1121 10:22:12.403771 20842 ts_tablet_manager.cc:1457] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:22:12.405230 20842 ts_tablet_manager.cc:1470] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5 I1121 10:22:12.405269 20842 log.cc:1153] T 8f0b8462a15040e88aebbbf5d5bf9dcd P 8754ab0c37934104af1213092861d2fa: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-0/wal/wals/8f0b8462a15040e88aebbbf5d5bf9dcd I1121 10:22:12.405649 19547 catalog_manager.cc:3512] TS 8754ab0c37934104af1213092861d2fa (127.5.70.1:33643): tablet 8f0b8462a15040e88aebbbf5d5bf9dcd (table rebalance_test_table_2 [id=302bc66d906a41448959053fb1bde011]) successfully deleted I1121 10:22:12.458937 20802 raft_consensus.cc:971] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: attempting to promote NON_VOTER 9a5dc7ff276144dfb28b69a0dd92218d to VOTER I1121 10:22:12.459118 20802 consensus_queue.cc:208] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } I1121 10:22:12.459558 20430 raft_consensus.cc:1181] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 LEARNER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.459578 19679 raft_consensus.cc:1181] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Refusing update from remote peer 4a73706e30e7445a93af7bd060d6103b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I1121 10:22:12.459764 20801 consensus_queue.cc:984] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.459847 20802 consensus_queue.cc:984] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I1121 10:22:12.460680 20802 raft_consensus.cc:2801] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.460886 19679 raft_consensus.cc:2801] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.461784 20430 raft_consensus.cc:2801] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.462154 19555 catalog_manager.cc:4143] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa reported cstate change: config changed from index 4 to 5, 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a73706e30e7445a93af7bd060d6103b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:12.491271 20062 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "b6bcafd84c0949b6b7d7fa7dc447f620" dest_uuid: "4a73706e30e7445a93af7bd060d6103b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:60176 I1121 10:22:12.491315 20062 raft_consensus.cc:581] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Received request to transfer leadership I1121 10:22:12.507359 20062 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "b6bcafd84c0949b6b7d7fa7dc447f620" dest_uuid: "4a73706e30e7445a93af7bd060d6103b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:60186 I1121 10:22:12.507397 20062 raft_consensus.cc:581] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Received request to transfer leadership I1121 10:22:12.713989 20062 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "b6bcafd84c0949b6b7d7fa7dc447f620" dest_uuid: "4a73706e30e7445a93af7bd060d6103b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:60190 I1121 10:22:12.714036 20062 raft_consensus.cc:581] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Received request to transfer leadership I1121 10:22:12.920951 20062 tablet_service.cc:1441] Received LeaderStepDown RPC: tablet_id: "b6bcafd84c0949b6b7d7fa7dc447f620" dest_uuid: "4a73706e30e7445a93af7bd060d6103b" mode: GRACEFUL from {username='slave'} at 127.0.0.1:60194 I1121 10:22:12.920994 20062 raft_consensus.cc:581] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Received request to transfer leadership I1121 10:22:13.048213 20165 raft_consensus.cc:900] Instructing follower 8754ab0c37934104af1213092861d2fa to start an election I1121 10:22:13.048319 20165 raft_consensus.cc:988] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Signalling peer 8754ab0c37934104af1213092861d2fato start an election I1121 10:22:13.048667 19679 tablet_service.cc:1413] Received Run Leader Election RPC: tablet_id: "b6bcafd84c0949b6b7d7fa7dc447f620" dest_uuid: "8754ab0c37934104af1213092861d2fa" from {username='slave'} at 127.5.70.4:39899 I1121 10:22:13.048703 19679 raft_consensus.cc:469] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Starting forced leader election (received explicit request) I1121 10:22:13.048712 19679 raft_consensus.cc:2905] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 1 FOLLOWER]: Advancing to term 2 I1121 10:22:13.049409 19679 raft_consensus.cc:491] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } I1121 10:22:13.049508 19679 leader_election.cc:296] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 2 election: Requested vote from peers 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823), 9a5dc7ff276144dfb28b69a0dd92218d (127.5.70.3:36993) I1121 10:22:13.049726 20062 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b6bcafd84c0949b6b7d7fa7dc447f620" candidate_uuid: "8754ab0c37934104af1213092861d2fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4a73706e30e7445a93af7bd060d6103b" I1121 10:22:13.049762 20062 raft_consensus.cc:2900] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Stepping down as leader of term 1 I1121 10:22:13.049772 20062 raft_consensus.cc:702] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a73706e30e7445a93af7bd060d6103b, State: Running, Role: LEADER I1121 10:22:13.049849 20062 consensus_queue.cc:231] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } I1121 10:22:13.049904 20062 raft_consensus.cc:2905] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 1 FOLLOWER]: Advancing to term 2 I1121 10:22:13.050565 20062 raft_consensus.cc:2359] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8754ab0c37934104af1213092861d2fa in term 2. I1121 10:22:13.050729 19618 leader_election.cc:310] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a73706e30e7445a93af7bd060d6103b, 8754ab0c37934104af1213092861d2fa; no voters: I1121 10:22:13.050817 20157 raft_consensus.cc:2653] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 2 FOLLOWER]: Leader election won for term 2 I1121 10:22:13.050892 20157 raft_consensus.cc:664] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 2 LEADER]: Becoming Leader. State: Replica: 8754ab0c37934104af1213092861d2fa, State: Running, Role: LEADER I1121 10:22:13.050974 20157 consensus_queue.cc:208] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } I1121 10:22:13.051542 19560 catalog_manager.cc:4143] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa reported cstate change: term changed from 1 to 2, leader changed from 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) to 8754ab0c37934104af1213092861d2fa (127.5.70.1). New cstate: current_term: 2 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I1121 10:22:13.053171 20430 tablet_service.cc:1296] Received RequestConsensusVote() RPC: tablet_id: "b6bcafd84c0949b6b7d7fa7dc447f620" candidate_uuid: "8754ab0c37934104af1213092861d2fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" I1121 10:22:13.053225 20430 raft_consensus.cc:2905] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 1 FOLLOWER]: Advancing to term 2 I1121 10:22:13.053813 20430 raft_consensus.cc:2359] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8754ab0c37934104af1213092861d2fa in term 2. I1121 10:22:13.544906 20062 raft_consensus.cc:1181] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 2 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I1121 10:22:13.545219 20157 consensus_queue.cc:984] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a73706e30e7445a93af7bd060d6103b" member_type: VOTER last_known_addr { host: "127.5.70.4" port: 44823 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:22:13.546331 20430 raft_consensus.cc:1181] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 2 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I1121 10:22:13.546568 20978 consensus_queue.cc:984] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I1121 10:22:14.132772 19679 consensus_queue.cc:208] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } I1121 10:22:14.133174 20430 raft_consensus.cc:1181] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 2 FOLLOWER]: Refusing update from remote peer 8754ab0c37934104af1213092861d2fa: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch) I1121 10:22:14.133457 21015 consensus_queue.cc:984] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I1121 10:22:14.134265 21015 raft_consensus.cc:2801] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:14.134519 20430 raft_consensus.cc:2801] T b6bcafd84c0949b6b7d7fa7dc447f620 P 9a5dc7ff276144dfb28b69a0dd92218d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } } } I1121 10:22:14.135300 19547 catalog_manager.cc:3678] ChangeConfig:REMOVE_PEER RPC for tablet b6bcafd84c0949b6b7d7fa7dc447f620 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I1121 10:22:14.135865 19560 catalog_manager.cc:4143] T b6bcafd84c0949b6b7d7fa7dc447f620 P 8754ab0c37934104af1213092861d2fa reported cstate change: config changed from index 5 to 7, VOTER 4a73706e30e7445a93af7bd060d6103b (127.5.70.4) evicted. New cstate: current_term: 2 leader_uuid: "8754ab0c37934104af1213092861d2fa" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8754ab0c37934104af1213092861d2fa" member_type: VOTER last_known_addr { host: "127.5.70.1" port: 33643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a5dc7ff276144dfb28b69a0dd92218d" member_type: VOTER last_known_addr { host: "127.5.70.3" port: 36993 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I1121 10:22:14.136859 20041 tablet_service.cc:1067] Processing DeleteTablet for tablet b6bcafd84c0949b6b7d7fa7dc447f620 with delete_type TABLET_DATA_TOMBSTONED (TS 4a73706e30e7445a93af7bd060d6103b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59982 I1121 10:22:14.137053 21021 tablet_replica.cc:282] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: stopping tablet replica I1121 10:22:14.137099 21021 raft_consensus.cc:2156] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 2 FOLLOWER]: Raft consensus shutting down. I1121 10:22:14.137117 21021 raft_consensus.cc:2183] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b [term 2 FOLLOWER]: Raft consensus is shut down! I1121 10:22:14.137403 21021 ts_tablet_manager.cc:1457] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I1121 10:22:14.139297 21021 ts_tablet_manager.cc:1470] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I1121 10:22:14.139333 21021 log.cc:1153] T b6bcafd84c0949b6b7d7fa7dc447f620 P 4a73706e30e7445a93af7bd060d6103b: Deleting WAL directory at /tmp/dist-test-taskdLJtk4/test-tmp/rebalancer_tool-test.1.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1574331670674946-5400/raft_consensus-itest-cluster/ts-3/wal/wals/b6bcafd84c0949b6b7d7fa7dc447f620 I1121 10:22:14.139632 19546 catalog_manager.cc:3512] TS 4a73706e30e7445a93af7bd060d6103b (127.5.70.4:44823): tablet b6bcafd84c0949b6b7d7fa7dc447f620 (table rebalance_test_table_0 [id=dca22bab1f104febb0f86c11e01c69b5]) successfully deleted I1121 10:22:14.277038 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 19607 I1121 10:22:14.280892 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 20229 I1121 10:22:14.284000 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 20353 I1121 10:22:14.287052 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-tserver with pid 19983 I1121 10:22:14.290345 5400 external_mini_cluster.cc:1237] Killing /tmp/dist-test-taskdLJtk4/build/release/bin/kudu-master with pid 19536 2019-11-21T10:22:14Z chronyd exiting [ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (9193 ms) [----------] 1 test from LocationAwareRebalancingParamTest (9193 ms total) [----------] Global test environment tear-down [==========] 6 tests from 6 test cases ran. (63632 ms total) [ PASSED ] 5 tests. [ FAILED ] 1 test, listed below: [ FAILED ] RebalanceIgnoredTserversTest.Basic 1 FAILED TEST